From 3d7715a0089edbb4089c8565ca40ff59e2aa504e Mon Sep 17 00:00:00 2001 From: Widthdom Date: Sun, 26 Jul 2026 16:55:57 +0900 Subject: [PATCH 01/30] Reduce masked-line reference allocations --- DEVELOPER_GUIDE.md | 11 +++ TESTING_GUIDE.md | 2 + .../+masked-reference-contexts.fixed.md | 21 ++++++ .../ReferenceExtractor.CoreReferenceLoop.cs | 14 ++++ tests/CodeIndex.Tests/PerformanceTests.cs | 69 +++++++++++++++++++ 5 files changed, 117 insertions(+) create mode 100644 changelog.d/unreleased/+masked-reference-contexts.fixed.md diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index 79db3cc4d4..960e4c0efa 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -509,6 +509,12 @@ string. `string.Split` creates an array and substrings for every import, dependency, path segment, or declaration item. Preserve the original empty-item, trimming, quote, and first-separator semantics when replacing it. +When structural masking turns a source line into whitespace, do not materialize +a trimmed copy merely to discover that the line has no references. Preserve +documentation handling and the build-automation and markup paths that inspect +the original line, but let ordinary C#, Java, and JavaScript / TypeScript code +paths skip masked multiline payloads before creating a reference context. + The C# value-receiver path is the reference example: local receiver scopes are derived from precomputed block spans for the containing function, and duplicate receiver records are tracked with a hash set. Regressions in this area should @@ -3494,6 +3500,11 @@ validation を source string 上で完結できる場合、index / span walk を import、dependency、path segment、declaration item ごとに array と substring を作る。置換時は 元の empty-item、trim、quote、first-separator semantics を維持する。 +構造マスクによって source line が空白だけになった場合、reference がないことを確認するため +だけに trim 済み copy を実体化してはならない。documentation handling と、original line を +検査する build-automation / markup 経路は維持しつつ、通常の C#、Java、 +JavaScript / TypeScript 経路では reference context を作る前に multiline payload を skip する。 + C# の value receiver 経路を参照例とする。local receiver の scope は containing function 用に 事前計算した block span から導出し、重複 receiver record は hash set で追跡する。この領域の regression には、scope rule の focused correctness test と、ユーザーが multi-hour indexing stall を diff --git a/TESTING_GUIDE.md b/TESTING_GUIDE.md index 50f8039a10..f53eac7f0f 100644 --- a/TESTING_GUIDE.md +++ b/TESTING_GUIDE.md @@ -576,6 +576,7 @@ Use `docs/test-doc-maintenance-plan.md` before moving oversized suites or adding Concurrent read and read-during-write scenarios (WAL mode validation), including the issue #180 bug-catching snapshot-isolation regressions for all three multi-statement reader entry points: (1) `GetStatus` seeds `refs == files * refsPerFile` and asserts every concurrent observation preserves that invariant; (2) `AnalyzeSymbol` seeds one symbol `S` plus matching reference/caller pairs, toggles a second file symmetrically, and asserts `references.Count == callers.Count` across every `inspect`/`analyze_symbol` bundle; (3) `GetRepoMap` seeds a baseline modified timestamp and toggles a newer file, asserting `latest_modified == workspace_latest_modified` across every map call. Each test fails without the DEFERRED-transaction wrap on the matching reader and passes with it. - `PerformanceTests.cs` Bounded CI smoke coverage plus large-scale data benchmarks. `CiPerformanceSmoke_IndexAndSearchSmallFixture_StaysWithinBudget` and the allocation budget guards run in the default `net8.0` suite, so they are blocking PR/CI checks on the production target, but their broad budgets are intended to catch only severe indexing/search or allocation regressions rather than act as benchmarks. `ReferenceExtraction_RepeatedSymbolMembership_StaysWithinAllocationBudget` uses dense C# private-property receivers and Python imported-type calls to prevent per-candidate full-symbol rescans from returning. `ReferenceExtraction_RepeatedContainerLookup_StaysWithinAllocationBudget` covers dense C# declaration containers and GitHub Actions jobs so name/range ownership resolution stays indexed. `Extraction_DenseDelimitedLists_StayWithinAllocationBudget` covers Python imports, YAML needs, JSON paths, and Fortran procedure lists without temporary split-array growth. `ReferenceDedupe_DenseLongIdentities_StayWithinAllocationBudget` keeps all-language dedupe identities value-based when qualified names are long. The 10K+ large-scale tests remain skip-by-default; run them manually with `--filter`. + `ReferenceExtraction_MaskedMultilinePayloads_StayWithinAllocationBudget` keeps C# raw strings, Java text blocks, and TypeScript template literals from materializing trimmed reference contexts after structural masking has made a line empty. `ReferenceExtraction_CSharpNoAliasDenseReferences_StaysWithinAllocationBudget` keeps 12,000 no-alias calls from paying for alias dedupe, `CSharpAliasCompaction_DenseDuplicates_StaysWithinAllocationBudget` guards pre-sized stable compaction after alias rewrites, and `MutualRecursion_DenseRepeatedQualifiedNames_StaysWithinAllocationBudget` covers repeated qualified C#- and Python-style cycle names without per-edge normalization strings. `ReusableStatSnapshot_OnePassMaterialization_StaysWithinAllocationBudget` warms a 1,024-row typed snapshot with its exact capacity, then prevents a second candidate collection or redundant path values from returning to the default `net8.0` path. `ReferenceBatchTransactions_RepositoryScaleAtomicFileScopeEliminatesControlledSqlScopes` models 321,352 references distributed over 856 files in the repository snapshot's five/six-batch shape and fixes the control-SQL contract at 5,009 public batch transaction scopes versus zero explicit atomic-file batch scopes; keep it deterministic and allocation-light instead of inserting all modeled rows or asserting elapsed time. For reference-line window performance audits, measure identical prebuilt reference rows against one-batch and 32-batch limits in alternating order, report both elapsed time and `GC.GetAllocatedBytesForCurrentThread`, and remove the timing harness after recording the result; end-to-end `--memory-trace` rebuilds remain corroborating evidence because extraction, graph finalization, and OS page-cache variance can dominate the persistence delta. @@ -1425,6 +1426,7 @@ dotnet test --filter "FullyQualifiedName~GitHelperTests" 並行読み取りと書き込み中読み取りシナリオ（WALモード検証）。issue #180 の bug-catching な snapshot 隔離回帰テストを 3 つの multi-statement reader 経路について含む。(1) `GetStatus` は `refs == files * refsPerFile` の seed 不変条件を立て、並行観測が常にこの条件を維持することを要求する。(2) `AnalyzeSymbol` はシンボル `S` に対して reference/caller を対称に 1 対 1 で seed し、もう 1 ファイルを対称に toggle することで `inspect` / `analyze_symbol` bundle の `references.Count == callers.Count` を常に保証する。(3) `GetRepoMap` はベースラインの modified と新しい toggle 対象ファイルを用意し、`latest_modified == workspace_latest_modified` が常に一致することを要求する。各テストは対応する reader の DEFERRED transaction を外すと落ち、戻すと通ることを確認済み。 - `PerformanceTests.cs` bounded な CI smoke と大規模データベンチマークを扱います。`CiPerformanceSmoke_IndexAndSearchSmallFixture_StaysWithinBudget` と allocation budget guard は通常の `net8.0` suite で実行されるため production target 上の PR / CI blocking check ですが、benchmark ではなく重大な indexing/search または allocation 退行だけを拾う広めの budget を使います。`ReferenceExtraction_RepeatedSymbolMembership_StaysWithinAllocationBudget` は密な C# private-property receiver と Python imported-type call を使い、candidate ごとの full-symbol 再走査が戻るのを防ぎます。`ReferenceExtraction_RepeatedContainerLookup_StaysWithinAllocationBudget` は密な C# declaration container と GitHub Actions job を扱い、name / range ownership 解決の索引化を維持します。`Extraction_DenseDelimitedLists_StayWithinAllocationBudget` は Python import、YAML needs、JSON path、Fortran procedure list を使い、一時 split-array の増加を防ぎます。`ReferenceDedupe_DenseLongIdentities_StayWithinAllocationBudget` は長い qualified name でも全言語共通 dedupe identity を value-based に維持します。10K+ の大規模テストは引き続きデフォルト Skip で、`--filter` で手動実行します。 + `ReferenceExtraction_MaskedMultilinePayloads_StayWithinAllocationBudget` は、構造マスク後に空行となった C# raw string、Java text block、TypeScript template literal から trim 済み reference context を実体化しないことを固定します。 `ReferenceExtraction_CSharpNoAliasDenseReferences_StaysWithinAllocationBudget` は alias のない 12,000 call が alias dedupe のコストを負わないこと、`CSharpAliasCompaction_DenseDuplicates_StaysWithinAllocationBudget` は alias rewrite 後の事前 capacity 付き stable compaction、`MutualRecursion_DenseRepeatedQualifiedNames_StaysWithinAllocationBudget` は C# / Python 形式の qualified cycle name が edge ごとの正規化文字列を作らないことを保証します。 `ReusableStatSnapshot_OnePassMaterialization_StaysWithinAllocationBudget` は exact capacity を渡した1,024行の typed snapshot を warm-up し、2つ目の候補 collection や重複 path value が通常の `net8.0` 経路へ戻らないようにします。 `ReferenceBatchTransactions_RepositoryScaleAtomicFileScopeEliminatesControlledSqlScopes` は321,352 refsを自己snapshotの5/6 batch形状で856 filesへ分配し、制御SQL契約をpublic batch transaction scope 5,009回対explicit atomic-file scope 0回に固定します。全model rowを挿入したり経過時間をassertしたりせず、deterministicでallocation-lightなまま維持してください。reference-line windowの性能監査では、同一の事前構築済みreference rowを1-batch上限と32-batch上限で交互に測り、経過時間と`GC.GetAllocatedBytesForCurrentThread`の両方を報告して、結果記録後にtiming harnessを削除します。end-to-endの`--memory-trace` rebuildは、extraction・graph finalize・OS page cacheの変動が永続化差を支配し得るため、補助証拠として扱ってください。 diff --git a/changelog.d/unreleased/+masked-reference-contexts.fixed.md b/changelog.d/unreleased/+masked-reference-contexts.fixed.md new file mode 100644 index 0000000000..101eaafab8 --- /dev/null +++ b/changelog.d/unreleased/+masked-reference-contexts.fixed.md @@ -0,0 +1,21 @@ +--- +category: fixed +affected: + - src/CodeIndex/Indexer/References/ReferenceExtractor.CoreReferenceLoop.cs + - tests/CodeIndex.Tests/PerformanceTests.cs + - DEVELOPER_GUIDE.md + - TESTING_GUIDE.md +--- + +## English + +- **Reduced reference-extraction allocations for large embedded multiline payloads** — + Structurally masked C# raw strings, Java text blocks, and JavaScript / + TypeScript template literals no longer materialize trimmed reference + contexts for lines that will be skipped. + +## 日本語 + +- **巨大な埋め込み multiline payload に対する reference extraction の allocation を削減しました** — + 構造マスク済みの C# raw string、Java text block、JavaScript / TypeScript template literal + では、skip される行の trim 済み reference context を実体化しません。 diff --git a/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreReferenceLoop.cs b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreReferenceLoop.cs index a150228026..af69bcecf8 100644 --- a/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreReferenceLoop.cs +++ b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreReferenceLoop.cs @@ -585,6 +585,20 @@ private static bool EmitCoreDocumentationAndSpecialLineReferences( ref state.PhpDocblockPropertyNames); } + if (string.IsNullOrWhiteSpace(preparedLine) + && request.Language + is not ("cmake" + or "justfile" + or "makefile" + or "msbuild" + or "graphql" + or "html" + or "markdown")) + { + sourceContext = string.Empty; + return false; + } + sourceContext = originalLine.Trim(); if (request.Language is "cmake" or "justfile" or "makefile" or "msbuild" diff --git a/tests/CodeIndex.Tests/PerformanceTests.cs b/tests/CodeIndex.Tests/PerformanceTests.cs index 2cb0f3297b..9dc79682b5 100644 --- a/tests/CodeIndex.Tests/PerformanceTests.cs +++ b/tests/CodeIndex.Tests/PerformanceTests.cs @@ -353,6 +353,75 @@ public void ReferenceExtraction_CsharpHotPath_StaysWithinAllocationBudget() Assert.True(allocatedBytes < 6_000_000, $"Reference extraction allocated {allocatedBytes:N0} bytes"); } +#if NET8_0 + [Fact] +#else + [Fact(Skip = PracticalBudgetTestTarget.SecondaryTargetSkipReason)] +#endif + public void ReferenceExtraction_MaskedMultilinePayloads_StayWithinAllocationBudget() + { + var payloadLine = $" {new string('x', 4_096)} TargetInsidePayload(); "; + var fixtures = new[] + { + ( + Language: "csharp", + Content: $$"""" + public sealed class C + { + private const string Payload = """ + {{string.Join('\n', Enumerable.Repeat(payloadLine, 256))}} + """; + public void Run() => CSharpTarget(); + } + """", + ExpectedTarget: "CSharpTarget"), + ( + Language: "java", + Content: $$"""" + public final class C { + private static final String PAYLOAD = """ + {{string.Join('\n', Enumerable.Repeat(payloadLine, 256))}} + """; + public void run() { JavaTarget(); } + } + """", + ExpectedTarget: "JavaTarget"), + ( + Language: "typescript", + Content: $$""" + const payload = ` + {{string.Join('\n', Enumerable.Repeat(payloadLine, 256))}} + `; + export function run() { TypeScriptTarget(); } + """, + ExpectedTarget: "TypeScriptTarget"), + }; + + long allocatedBytes = 0; + foreach (var fixture in fixtures) + { + var symbols = SymbolExtractor.Extract(1, fixture.Language, fixture.Content); + _ = ReferenceExtractor.Extract(1, fixture.Language, fixture.Content, symbols); + + List? references = null; + allocatedBytes += MeasureAllocatedBytes( + () => references = ReferenceExtractor.Extract( + 1, + fixture.Language, + fixture.Content, + symbols)); + + Assert.Contains( + references!, + reference => reference.ReferenceKind == "call" + && reference.SymbolName == fixture.ExpectedTarget); + } + + Assert.True( + allocatedBytes < 23_000_000, + $"Masked multiline payload extraction allocated {allocatedBytes:N0} bytes"); + } + #if NET8_0 [Fact] #else From 04ec0b282b0fae3db31989503fde98ee82396dbe Mon Sep 17 00:00:00 2001 From: Widthdom Date: Sun, 26 Jul 2026 17:01:19 +0900 Subject: [PATCH 02/30] Scan C++ header samples without line arrays --- DEVELOPER_GUIDE.md | 9 ++++++++ TESTING_GUIDE.md | 2 ++ .../unreleased/+cpp-header-line-walk.fixed.md | 20 ++++++++++++++++++ .../Scanning/FileIndexer.LanguageDetection.cs | 21 +++++++++++++++---- tests/CodeIndex.Tests/PerformanceTests.cs | 21 +++++++++++++++++++ 5 files changed, 69 insertions(+), 4 deletions(-) create mode 100644 changelog.d/unreleased/+cpp-header-line-walk.fixed.md diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index 960e4c0efa..5369af9c67 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -515,6 +515,11 @@ documentation handling and the build-automation and markup paths that inspect the original line, but let ordinary C#, Java, and JavaScript / TypeScript code paths skip masked multiline payloads before creating a reference context. +C / C++ header disambiguation operates on bounded lexical samples. Walk those +samples with spans and newline indices; splitting a sample into a line array +temporarily duplicates every sampled line and scales poorly across large +repositories with many ambiguous `.h` files. + The C# value-receiver path is the reference example: local receiver scopes are derived from precomputed block spans for the containing function, and duplicate receiver records are tracked with a hash set. Regressions in this area should @@ -3505,6 +3510,10 @@ import、dependency、path segment、declaration item ごとに array と substr 検査する build-automation / markup 経路は維持しつつ、通常の C#、Java、 JavaScript / TypeScript 経路では reference context を作る前に multiline payload を skip する。 +C / C++ header の曖昧性解決は bounded lexical sample 上で行う。sample は span と newline index +で走査し、line array に split してはならない。split は sample 内の全行を一時的に複製し、 +曖昧な `.h` file が多い巨大 repository でスケールしにくい。 + C# の value receiver 経路を参照例とする。local receiver の scope は containing function 用に 事前計算した block span から導出し、重複 receiver record は hash set で追跡する。この領域の regression には、scope rule の focused correctness test と、ユーザーが multi-hour indexing stall を diff --git a/TESTING_GUIDE.md b/TESTING_GUIDE.md index f53eac7f0f..15ae782fce 100644 --- a/TESTING_GUIDE.md +++ b/TESTING_GUIDE.md @@ -577,6 +577,7 @@ Use `docs/test-doc-maintenance-plan.md` before moving oversized suites or adding - `PerformanceTests.cs` Bounded CI smoke coverage plus large-scale data benchmarks. `CiPerformanceSmoke_IndexAndSearchSmallFixture_StaysWithinBudget` and the allocation budget guards run in the default `net8.0` suite, so they are blocking PR/CI checks on the production target, but their broad budgets are intended to catch only severe indexing/search or allocation regressions rather than act as benchmarks. `ReferenceExtraction_RepeatedSymbolMembership_StaysWithinAllocationBudget` uses dense C# private-property receivers and Python imported-type calls to prevent per-candidate full-symbol rescans from returning. `ReferenceExtraction_RepeatedContainerLookup_StaysWithinAllocationBudget` covers dense C# declaration containers and GitHub Actions jobs so name/range ownership resolution stays indexed. `Extraction_DenseDelimitedLists_StayWithinAllocationBudget` covers Python imports, YAML needs, JSON paths, and Fortran procedure lists without temporary split-array growth. `ReferenceDedupe_DenseLongIdentities_StayWithinAllocationBudget` keeps all-language dedupe identities value-based when qualified names are long. The 10K+ large-scale tests remain skip-by-default; run them manually with `--filter`. `ReferenceExtraction_MaskedMultilinePayloads_StayWithinAllocationBudget` keeps C# raw strings, Java text blocks, and TypeScript template literals from materializing trimmed reference contexts after structural masking has made a line empty. + `CppHeaderDetection_LargeSample_DoesNotMaterializeLineArrays` keeps bounded C / C++ header-disambiguation samples on span-based line walks instead of allocating a string and array for every sampled line. `ReferenceExtraction_CSharpNoAliasDenseReferences_StaysWithinAllocationBudget` keeps 12,000 no-alias calls from paying for alias dedupe, `CSharpAliasCompaction_DenseDuplicates_StaysWithinAllocationBudget` guards pre-sized stable compaction after alias rewrites, and `MutualRecursion_DenseRepeatedQualifiedNames_StaysWithinAllocationBudget` covers repeated qualified C#- and Python-style cycle names without per-edge normalization strings. `ReusableStatSnapshot_OnePassMaterialization_StaysWithinAllocationBudget` warms a 1,024-row typed snapshot with its exact capacity, then prevents a second candidate collection or redundant path values from returning to the default `net8.0` path. `ReferenceBatchTransactions_RepositoryScaleAtomicFileScopeEliminatesControlledSqlScopes` models 321,352 references distributed over 856 files in the repository snapshot's five/six-batch shape and fixes the control-SQL contract at 5,009 public batch transaction scopes versus zero explicit atomic-file batch scopes; keep it deterministic and allocation-light instead of inserting all modeled rows or asserting elapsed time. For reference-line window performance audits, measure identical prebuilt reference rows against one-batch and 32-batch limits in alternating order, report both elapsed time and `GC.GetAllocatedBytesForCurrentThread`, and remove the timing harness after recording the result; end-to-end `--memory-trace` rebuilds remain corroborating evidence because extraction, graph finalization, and OS page-cache variance can dominate the persistence delta. @@ -1427,6 +1428,7 @@ dotnet test --filter "FullyQualifiedName~GitHelperTests" - `PerformanceTests.cs` bounded な CI smoke と大規模データベンチマークを扱います。`CiPerformanceSmoke_IndexAndSearchSmallFixture_StaysWithinBudget` と allocation budget guard は通常の `net8.0` suite で実行されるため production target 上の PR / CI blocking check ですが、benchmark ではなく重大な indexing/search または allocation 退行だけを拾う広めの budget を使います。`ReferenceExtraction_RepeatedSymbolMembership_StaysWithinAllocationBudget` は密な C# private-property receiver と Python imported-type call を使い、candidate ごとの full-symbol 再走査が戻るのを防ぎます。`ReferenceExtraction_RepeatedContainerLookup_StaysWithinAllocationBudget` は密な C# declaration container と GitHub Actions job を扱い、name / range ownership 解決の索引化を維持します。`Extraction_DenseDelimitedLists_StayWithinAllocationBudget` は Python import、YAML needs、JSON path、Fortran procedure list を使い、一時 split-array の増加を防ぎます。`ReferenceDedupe_DenseLongIdentities_StayWithinAllocationBudget` は長い qualified name でも全言語共通 dedupe identity を value-based に維持します。10K+ の大規模テストは引き続きデフォルト Skip で、`--filter` で手動実行します。 `ReferenceExtraction_MaskedMultilinePayloads_StayWithinAllocationBudget` は、構造マスク後に空行となった C# raw string、Java text block、TypeScript template literal から trim 済み reference context を実体化しないことを固定します。 + `CppHeaderDetection_LargeSample_DoesNotMaterializeLineArrays` は、bounded な C / C++ header 判定 sample を span ベースで行走査し、sampled line ごとの string と array を割り当てないことを固定します。 `ReferenceExtraction_CSharpNoAliasDenseReferences_StaysWithinAllocationBudget` は alias のない 12,000 call が alias dedupe のコストを負わないこと、`CSharpAliasCompaction_DenseDuplicates_StaysWithinAllocationBudget` は alias rewrite 後の事前 capacity 付き stable compaction、`MutualRecursion_DenseRepeatedQualifiedNames_StaysWithinAllocationBudget` は C# / Python 形式の qualified cycle name が edge ごとの正規化文字列を作らないことを保証します。 `ReusableStatSnapshot_OnePassMaterialization_StaysWithinAllocationBudget` は exact capacity を渡した1,024行の typed snapshot を warm-up し、2つ目の候補 collection や重複 path value が通常の `net8.0` 経路へ戻らないようにします。 `ReferenceBatchTransactions_RepositoryScaleAtomicFileScopeEliminatesControlledSqlScopes` は321,352 refsを自己snapshotの5/6 batch形状で856 filesへ分配し、制御SQL契約をpublic batch transaction scope 5,009回対explicit atomic-file scope 0回に固定します。全model rowを挿入したり経過時間をassertしたりせず、deterministicでallocation-lightなまま維持してください。reference-line windowの性能監査では、同一の事前構築済みreference rowを1-batch上限と32-batch上限で交互に測り、経過時間と`GC.GetAllocatedBytesForCurrentThread`の両方を報告して、結果記録後にtiming harnessを削除します。end-to-endの`--memory-trace` rebuildは、extraction・graph finalize・OS page cacheの変動が永続化差を支配し得るため、補助証拠として扱ってください。 diff --git a/changelog.d/unreleased/+cpp-header-line-walk.fixed.md b/changelog.d/unreleased/+cpp-header-line-walk.fixed.md new file mode 100644 index 0000000000..3faaa0d12b --- /dev/null +++ b/changelog.d/unreleased/+cpp-header-line-walk.fixed.md @@ -0,0 +1,20 @@ +--- +category: fixed +affected: + - src/CodeIndex/Indexer/Scanning/FileIndexer.LanguageDetection.cs + - tests/CodeIndex.Tests/PerformanceTests.cs + - DEVELOPER_GUIDE.md + - TESTING_GUIDE.md +--- + +## English + +- **Reduced C / C++ header-detection allocations for large repositories** — + Bounded lexical samples for ambiguous `.h` files are now scanned by newline + index instead of materializing an array and a string for every sampled line. + +## 日本語 + +- **巨大 repository の C / C++ header 判定 allocation を削減しました** — + 曖昧な `.h` file の bounded lexical sample を、sampled line ごとの array と string + に実体化せず newline index で走査するようにしました。 diff --git a/src/CodeIndex/Indexer/Scanning/FileIndexer.LanguageDetection.cs b/src/CodeIndex/Indexer/Scanning/FileIndexer.LanguageDetection.cs index cf668f8770..3ab1175743 100644 --- a/src/CodeIndex/Indexer/Scanning/FileIndexer.LanguageDetection.cs +++ b/src/CodeIndex/Indexer/Scanning/FileIndexer.LanguageDetection.cs @@ -789,20 +789,33 @@ private static CppHeaderDetectionResult DetectCppHeaderLanguage(string content) private static bool ContainsCppHeaderMarker(string content, bool firstLineIsComplete) { - var lines = content.Split('\n'); - - for (var lineIndex = 0; lineIndex < lines.Length; lineIndex++) + var lineIndex = 0; + var lineStart = 0; + while (lineStart <= content.Length) { - var line = lines[lineIndex].AsSpan(); + var lineBreak = content.IndexOf('\n', lineStart); + var lineEnd = lineBreak >= 0 ? lineBreak : content.Length; + var line = content.AsSpan(lineStart, lineEnd - lineStart); if (line.Length > 0 && line[^1] == '\r') line = line[..^1]; if (LooksLikeCppHeaderLine(line, allowLineStartMarkers: lineIndex > 0 || firstLineIsComplete)) return true; + + if (lineBreak < 0) + break; + + lineStart = lineBreak + 1; + lineIndex++; } return false; } + internal static bool ContainsCppHeaderMarkerForTesting( + string content, + bool firstLineIsComplete = true) + => ContainsCppHeaderMarker(content, firstLineIsComplete); + private static bool IsCppLogicalLineBoundary(string content, int index) { if (index == 0) diff --git a/tests/CodeIndex.Tests/PerformanceTests.cs b/tests/CodeIndex.Tests/PerformanceTests.cs index 9dc79682b5..cf0c1c1a62 100644 --- a/tests/CodeIndex.Tests/PerformanceTests.cs +++ b/tests/CodeIndex.Tests/PerformanceTests.cs @@ -422,6 +422,27 @@ public final class C { $"Masked multiline payload extraction allocated {allocatedBytes:N0} bytes"); } +#if NET8_0 + [Fact] +#else + [Fact(Skip = PracticalBudgetTestTarget.SecondaryTargetSkipReason)] +#endif + public void CppHeaderDetection_LargeSample_DoesNotMaterializeLineArrays() + { + var content = string.Join( + '\n', + Enumerable.Range(0, 8_192) + .Select(index => $"struct record_{index} {{ int value; }};")); + _ = FileIndexer.ContainsCppHeaderMarkerForTesting(content); + + var allocatedBytes = MeasureAllocatedBytes( + () => FileIndexer.ContainsCppHeaderMarkerForTesting(content)); + + Assert.True( + allocatedBytes < 1_024, + $"C/C++ header detection allocated {allocatedBytes:N0} bytes"); + } + #if NET8_0 [Fact] #else From ba009a946bd993db5826038d29acd90ab949bd61 Mon Sep 17 00:00:00 2001 From: Widthdom Date: Sun, 26 Jul 2026 17:08:27 +0900 Subject: [PATCH 03/30] Walk delimited extractor inputs with spans --- DEVELOPER_GUIDE.md | 6 ++ TESTING_GUIDE.md | 2 + .../+delimited-extractor-spans.fixed.md | 27 ++++++ .../Indexer/DelimitedSpanEnumerable.cs | 84 +++++++++++++++++++ .../Languages/HdlReferenceExtractor.Scopes.cs | 10 ++- .../Languages/HdlReferenceExtractor.cs | 48 ++++++++--- .../RepositoryMetadataReferenceExtractor.cs | 23 ++++- .../Languages/ShaderReferenceExtractor.cs | 12 ++- .../SymbolExtractor.RepositoryMetadata.cs | 40 ++++++++- tests/CodeIndex.Tests/PerformanceTests.cs | 42 ++++++++++ 10 files changed, 271 insertions(+), 23 deletions(-) create mode 100644 changelog.d/unreleased/+delimited-extractor-spans.fixed.md create mode 100644 src/CodeIndex/Indexer/DelimitedSpanEnumerable.cs diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index 5369af9c67..2f88605cae 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -508,6 +508,9 @@ consumer needs only one item at a time or validation can stay on the source string. `string.Split` creates an array and substrings for every import, dependency, path segment, or declaration item. Preserve the original empty-item, trimming, quote, and first-separator semantics when replacing it. +`DelimitedSpanEnumerable` is the shared allocation-free walker for single +delimiters; repository metadata, application manifests, VHDL declarations and +package paths, and CUDA parameter headers use it instead of split arrays. When structural masking turns a source line into whitespace, do not materialize a trimmed copy merely to discover that the line has no references. Preserve @@ -3504,6 +3507,9 @@ hot extractor の delimiter-only parsing では、consumer が item を一度に validation を source string 上で完結できる場合、index / span walk を優先する。`string.Split` は import、dependency、path segment、declaration item ごとに array と substring を作る。置換時は 元の empty-item、trim、quote、first-separator semantics を維持する。 +single delimiter には allocation-free な共通 walker `DelimitedSpanEnumerable` を使う。 +repository metadata、application manifest、VHDL declaration / package path、CUDA parameter +header は split array を作らずこの walker で処理する。 構造マスクによって source line が空白だけになった場合、reference がないことを確認するため だけに trim 済み copy を実体化してはならない。documentation handling と、original line を diff --git a/TESTING_GUIDE.md b/TESTING_GUIDE.md index 15ae782fce..1810bcdbce 100644 --- a/TESTING_GUIDE.md +++ b/TESTING_GUIDE.md @@ -578,6 +578,7 @@ Use `docs/test-doc-maintenance-plan.md` before moving oversized suites or adding Bounded CI smoke coverage plus large-scale data benchmarks. `CiPerformanceSmoke_IndexAndSearchSmallFixture_StaysWithinBudget` and the allocation budget guards run in the default `net8.0` suite, so they are blocking PR/CI checks on the production target, but their broad budgets are intended to catch only severe indexing/search or allocation regressions rather than act as benchmarks. `ReferenceExtraction_RepeatedSymbolMembership_StaysWithinAllocationBudget` uses dense C# private-property receivers and Python imported-type calls to prevent per-candidate full-symbol rescans from returning. `ReferenceExtraction_RepeatedContainerLookup_StaysWithinAllocationBudget` covers dense C# declaration containers and GitHub Actions jobs so name/range ownership resolution stays indexed. `Extraction_DenseDelimitedLists_StayWithinAllocationBudget` covers Python imports, YAML needs, JSON paths, and Fortran procedure lists without temporary split-array growth. `ReferenceDedupe_DenseLongIdentities_StayWithinAllocationBudget` keeps all-language dedupe identities value-based when qualified names are long. The 10K+ large-scale tests remain skip-by-default; run them manually with `--filter`. `ReferenceExtraction_MaskedMultilinePayloads_StayWithinAllocationBudget` keeps C# raw strings, Java text blocks, and TypeScript template literals from materializing trimmed reference contexts after structural masking has made a line empty. `CppHeaderDetection_LargeSample_DoesNotMaterializeLineArrays` keeps bounded C / C++ header-disambiguation samples on span-based line walks instead of allocating a string and array for every sampled line. + `DelimitedSpanWalking_DenseExtractorLists_DoesNotAllocate` locks the shared single-delimiter walker to allocation-free trim/remove-empty semantics used by repository metadata, application manifests, VHDL, and CUDA extraction. `ReferenceExtraction_CSharpNoAliasDenseReferences_StaysWithinAllocationBudget` keeps 12,000 no-alias calls from paying for alias dedupe, `CSharpAliasCompaction_DenseDuplicates_StaysWithinAllocationBudget` guards pre-sized stable compaction after alias rewrites, and `MutualRecursion_DenseRepeatedQualifiedNames_StaysWithinAllocationBudget` covers repeated qualified C#- and Python-style cycle names without per-edge normalization strings. `ReusableStatSnapshot_OnePassMaterialization_StaysWithinAllocationBudget` warms a 1,024-row typed snapshot with its exact capacity, then prevents a second candidate collection or redundant path values from returning to the default `net8.0` path. `ReferenceBatchTransactions_RepositoryScaleAtomicFileScopeEliminatesControlledSqlScopes` models 321,352 references distributed over 856 files in the repository snapshot's five/six-batch shape and fixes the control-SQL contract at 5,009 public batch transaction scopes versus zero explicit atomic-file batch scopes; keep it deterministic and allocation-light instead of inserting all modeled rows or asserting elapsed time. For reference-line window performance audits, measure identical prebuilt reference rows against one-batch and 32-batch limits in alternating order, report both elapsed time and `GC.GetAllocatedBytesForCurrentThread`, and remove the timing harness after recording the result; end-to-end `--memory-trace` rebuilds remain corroborating evidence because extraction, graph finalization, and OS page-cache variance can dominate the persistence delta. @@ -1429,6 +1430,7 @@ dotnet test --filter "FullyQualifiedName~GitHelperTests" bounded な CI smoke と大規模データベンチマークを扱います。`CiPerformanceSmoke_IndexAndSearchSmallFixture_StaysWithinBudget` と allocation budget guard は通常の `net8.0` suite で実行されるため production target 上の PR / CI blocking check ですが、benchmark ではなく重大な indexing/search または allocation 退行だけを拾う広めの budget を使います。`ReferenceExtraction_RepeatedSymbolMembership_StaysWithinAllocationBudget` は密な C# private-property receiver と Python imported-type call を使い、candidate ごとの full-symbol 再走査が戻るのを防ぎます。`ReferenceExtraction_RepeatedContainerLookup_StaysWithinAllocationBudget` は密な C# declaration container と GitHub Actions job を扱い、name / range ownership 解決の索引化を維持します。`Extraction_DenseDelimitedLists_StayWithinAllocationBudget` は Python import、YAML needs、JSON path、Fortran procedure list を使い、一時 split-array の増加を防ぎます。`ReferenceDedupe_DenseLongIdentities_StayWithinAllocationBudget` は長い qualified name でも全言語共通 dedupe identity を value-based に維持します。10K+ の大規模テストは引き続きデフォルト Skip で、`--filter` で手動実行します。 `ReferenceExtraction_MaskedMultilinePayloads_StayWithinAllocationBudget` は、構造マスク後に空行となった C# raw string、Java text block、TypeScript template literal から trim 済み reference context を実体化しないことを固定します。 `CppHeaderDetection_LargeSample_DoesNotMaterializeLineArrays` は、bounded な C / C++ header 判定 sample を span ベースで行走査し、sampled line ごとの string と array を割り当てないことを固定します。 + `DelimitedSpanWalking_DenseExtractorLists_DoesNotAllocate` は、repository metadata、application manifest、VHDL、CUDA extraction が共有する single-delimiter walker の trim / remove-empty semantics を allocation-free に固定します。 `ReferenceExtraction_CSharpNoAliasDenseReferences_StaysWithinAllocationBudget` は alias のない 12,000 call が alias dedupe のコストを負わないこと、`CSharpAliasCompaction_DenseDuplicates_StaysWithinAllocationBudget` は alias rewrite 後の事前 capacity 付き stable compaction、`MutualRecursion_DenseRepeatedQualifiedNames_StaysWithinAllocationBudget` は C# / Python 形式の qualified cycle name が edge ごとの正規化文字列を作らないことを保証します。 `ReusableStatSnapshot_OnePassMaterialization_StaysWithinAllocationBudget` は exact capacity を渡した1,024行の typed snapshot を warm-up し、2つ目の候補 collection や重複 path value が通常の `net8.0` 経路へ戻らないようにします。 `ReferenceBatchTransactions_RepositoryScaleAtomicFileScopeEliminatesControlledSqlScopes` は321,352 refsを自己snapshotの5/6 batch形状で856 filesへ分配し、制御SQL契約をpublic batch transaction scope 5,009回対explicit atomic-file scope 0回に固定します。全model rowを挿入したり経過時間をassertしたりせず、deterministicでallocation-lightなまま維持してください。reference-line windowの性能監査では、同一の事前構築済みreference rowを1-batch上限と32-batch上限で交互に測り、経過時間と`GC.GetAllocatedBytesForCurrentThread`の両方を報告して、結果記録後にtiming harnessを削除します。end-to-endの`--memory-trace` rebuildは、extraction・graph finalize・OS page cacheの変動が永続化差を支配し得るため、補助証拠として扱ってください。 diff --git a/changelog.d/unreleased/+delimited-extractor-spans.fixed.md b/changelog.d/unreleased/+delimited-extractor-spans.fixed.md new file mode 100644 index 0000000000..41613cfc88 --- /dev/null +++ b/changelog.d/unreleased/+delimited-extractor-spans.fixed.md @@ -0,0 +1,27 @@ +--- +category: fixed +affected: + - src/CodeIndex/Indexer/DelimitedSpanEnumerable.cs + - src/CodeIndex/Indexer/Symbols/SymbolExtractor.RepositoryMetadata.cs + - src/CodeIndex/Indexer/References/Languages/RepositoryMetadataReferenceExtractor.cs + - src/CodeIndex/Indexer/References/Languages/HdlReferenceExtractor.cs + - src/CodeIndex/Indexer/References/Languages/HdlReferenceExtractor.Scopes.cs + - src/CodeIndex/Indexer/References/Languages/ShaderReferenceExtractor.cs + - tests/CodeIndex.Tests/PerformanceTests.cs + - DEVELOPER_GUIDE.md + - TESTING_GUIDE.md +--- + +## English + +- **Removed split-array growth from delimiter-only extractor paths** — + Repository metadata, application manifests, VHDL declarations and package + imports, and CUDA kernel parameters now share an allocation-free span walker + while preserving trimming and empty-entry behavior. + +## 日本語 + +- **delimiter-only extractor 経路の split-array 増加を解消しました** — + repository metadata、application manifest、VHDL declaration / package import、 + CUDA kernel parameter は、trim と empty-entry の意味論を維持しつつ allocation-free + な span walker を共有します。 diff --git a/src/CodeIndex/Indexer/DelimitedSpanEnumerable.cs b/src/CodeIndex/Indexer/DelimitedSpanEnumerable.cs new file mode 100644 index 0000000000..28e7f9fa01 --- /dev/null +++ b/src/CodeIndex/Indexer/DelimitedSpanEnumerable.cs @@ -0,0 +1,84 @@ +namespace CodeIndex.Indexer; + +internal readonly ref struct DelimitedSpanEnumerable +{ + private readonly ReadOnlySpan _value; + private readonly char _delimiter; + private readonly bool _trimEntries; + private readonly bool _removeEmptyEntries; + + internal DelimitedSpanEnumerable( + ReadOnlySpan value, + char delimiter, + bool trimEntries = false, + bool removeEmptyEntries = false) + { + _value = value; + _delimiter = delimiter; + _trimEntries = trimEntries; + _removeEmptyEntries = removeEmptyEntries; + } + + public Enumerator GetEnumerator() => + new(_value, _delimiter, _trimEntries, _removeEmptyEntries); + + internal ref struct Enumerator + { + private readonly ReadOnlySpan _value; + private readonly char _delimiter; + private readonly bool _trimEntries; + private readonly bool _removeEmptyEntries; + private int _nextStart; + private bool _finished; + + internal Enumerator( + ReadOnlySpan value, + char delimiter, + bool trimEntries, + bool removeEmptyEntries) + { + _value = value; + _delimiter = delimiter; + _trimEntries = trimEntries; + _removeEmptyEntries = removeEmptyEntries; + _nextStart = 0; + _finished = false; + Current = default; + CurrentStart = 0; + } + + public ReadOnlySpan Current { get; private set; } + public int CurrentStart { get; private set; } + + public bool MoveNext() + { + while (!_finished) + { + var start = _nextStart; + var relativeEnd = _value[start..].IndexOf(_delimiter); + var end = relativeEnd < 0 + ? _value.Length + : start + relativeEnd; + _finished = relativeEnd < 0; + _nextStart = end + 1; + + var entry = _value[start..end]; + if (_trimEntries) + { + var trimmed = entry.Trim(); + start += entry.Length - entry.TrimStart().Length; + entry = trimmed; + } + + if (_removeEmptyEntries && entry.IsEmpty) + continue; + + Current = entry; + CurrentStart = start; + return true; + } + + return false; + } + } +} diff --git a/src/CodeIndex/Indexer/References/Languages/HdlReferenceExtractor.Scopes.cs b/src/CodeIndex/Indexer/References/Languages/HdlReferenceExtractor.Scopes.cs index 128c144911..e04f1b6ee0 100644 --- a/src/CodeIndex/Indexer/References/Languages/HdlReferenceExtractor.Scopes.cs +++ b/src/CodeIndex/Indexer/References/Languages/HdlReferenceExtractor.Scopes.cs @@ -345,8 +345,14 @@ private static bool TryGetVhdlDesignUnitKey(string line, out string key) private static void AddVhdlDeclaredNames(HashSet names, string value) { - foreach (var name in value.Split(',', StringSplitOptions.TrimEntries | StringSplitOptions.RemoveEmptyEntries)) - names.Add(name); + foreach (var name in new DelimitedSpanEnumerable( + value.AsSpan(), + ',', + trimEntries: true, + removeEmptyEntries: true)) + { + names.Add(name.ToString()); + } } private static string NormalizeVerilogScopeKind(string kind) diff --git a/src/CodeIndex/Indexer/References/Languages/HdlReferenceExtractor.cs b/src/CodeIndex/Indexer/References/Languages/HdlReferenceExtractor.cs index 7cd1e2723f..96d11820b5 100644 --- a/src/CodeIndex/Indexer/References/Languages/HdlReferenceExtractor.cs +++ b/src/CodeIndex/Indexer/References/Languages/HdlReferenceExtractor.cs @@ -605,18 +605,44 @@ private static void EmitVhdlStructuralReferences( private static (string PackageName, int Offset) SelectVhdlPackage(string path) { - var components = path.Split('.'); - var packageIndex = components.Length switch + var componentCount = 0; + var previousComponent = ReadOnlySpan.Empty; + var previousOffset = 0; + var lastComponent = ReadOnlySpan.Empty; + var lastOffset = 0; + foreach (var component in new DelimitedSpanEnumerable(path.AsSpan(), '.')) { - 1 => 0, - 2 when string.Equals(components[1], "all", StringComparison.OrdinalIgnoreCase) => 0, - 2 => 1, - _ => components.Length - 2, - }; - var offset = 0; - for (var index = 0; index < packageIndex; index++) - offset += components[index].Length + 1; - return (components[packageIndex], offset); + previousComponent = lastComponent; + previousOffset = lastOffset; + lastComponent = component; + lastOffset = componentCount == 0 + ? 0 + : lastOffset + previousComponent.Length + 1; + componentCount++; + } + + ReadOnlySpan package; + int offset; + switch (componentCount) + { + case 1: + package = lastComponent; + offset = lastOffset; + break; + case 2 when lastComponent.Equals("all", StringComparison.OrdinalIgnoreCase): + package = previousComponent; + offset = previousOffset; + break; + case 2: + package = lastComponent; + offset = lastOffset; + break; + default: + package = previousComponent; + offset = previousOffset; + break; + } + return (package.ToString(), offset); } private static void EmitKnownHdlReferences( diff --git a/src/CodeIndex/Indexer/References/Languages/RepositoryMetadataReferenceExtractor.cs b/src/CodeIndex/Indexer/References/Languages/RepositoryMetadataReferenceExtractor.cs index 1a9e198029..de1ab8b355 100644 --- a/src/CodeIndex/Indexer/References/Languages/RepositoryMetadataReferenceExtractor.cs +++ b/src/CodeIndex/Indexer/References/Languages/RepositoryMetadataReferenceExtractor.cs @@ -230,8 +230,21 @@ internal static List ExtractApplicationManifest( var privatePath = reader.GetAttribute("privatePath"); if (!string.IsNullOrWhiteSpace(privatePath)) { - foreach (var path in privatePath.Split(';', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries)) - AddManifestPath(references, seen, fileId, path, context, lineNumber, manifestContainer); + foreach (var path in new DelimitedSpanEnumerable( + privatePath.AsSpan(), + ';', + trimEntries: true, + removeEmptyEntries: true)) + { + AddManifestPath( + references, + seen, + fileId, + path.ToString(), + context, + lineNumber, + manifestContainer); + } } } @@ -520,9 +533,11 @@ private static bool TryNormalizeRepositoryPath( return false; } - foreach (var segment in value.Split('/')) + foreach (var segment in new DelimitedSpanEnumerable(value.AsSpan(), '/')) { - if (segment.Length == 0 || segment is "." or "..") + if (segment.IsEmpty + || segment.SequenceEqual(".") + || segment.SequenceEqual("..")) return false; } diff --git a/src/CodeIndex/Indexer/References/Languages/ShaderReferenceExtractor.cs b/src/CodeIndex/Indexer/References/Languages/ShaderReferenceExtractor.cs index 6114356d12..2cdf80a9d4 100644 --- a/src/CodeIndex/Indexer/References/Languages/ShaderReferenceExtractor.cs +++ b/src/CodeIndex/Indexer/References/Languages/ShaderReferenceExtractor.cs @@ -275,9 +275,17 @@ private static void TrackCudaKernelParameters( if (!match.Success) continue; - foreach (var parameter in match.Groups["parameters"].Value.Split(',')) + var parameters = match.Groups["parameters"]; + var parameterEnumerator = new DelimitedSpanEnumerable( + header.AsSpan(parameters.Index, parameters.Length), + ',').GetEnumerator(); + while (parameterEnumerator.MoveNext()) { - var name = CudaParameterNameRegex.Match(parameter).Groups["name"]; + var parameter = parameterEnumerator.Current; + var name = CudaParameterNameRegex.Match( + header, + parameters.Index + parameterEnumerator.CurrentStart, + parameter.Length).Groups["name"]; if (!name.Success || string.IsNullOrWhiteSpace(name.Value)) continue; diff --git a/src/CodeIndex/Indexer/Symbols/SymbolExtractor.RepositoryMetadata.cs b/src/CodeIndex/Indexer/Symbols/SymbolExtractor.RepositoryMetadata.cs index a26cfe676d..42b5d85634 100644 --- a/src/CodeIndex/Indexer/Symbols/SymbolExtractor.RepositoryMetadata.cs +++ b/src/CodeIndex/Indexer/Symbols/SymbolExtractor.RepositoryMetadata.cs @@ -347,16 +347,48 @@ internal static bool TryGetGitAttributesTokens( pattern = trimmed[..patternEnd].ToString(); } - var tokens = trimmed[patternEnd..].ToString().Split( - [' ', '\t'], - StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries); - if (tokens.Length == 0) + var attributesSpan = trimmed[patternEnd..]; + var attributeCount = CountGitAttributeTokens(attributesSpan); + if (attributeCount == 0) return false; + var tokens = new string[attributeCount]; + var tokenIndex = 0; + while (!attributesSpan.IsEmpty) + { + var separator = attributesSpan.IndexOfAny(' ', '\t'); + var token = separator < 0 ? attributesSpan : attributesSpan[..separator]; + token = token.Trim(); + if (!token.IsEmpty) + tokens[tokenIndex++] = token.ToString(); + + if (separator < 0) + break; + attributesSpan = attributesSpan[(separator + 1)..]; + } + attributes = tokens; return true; } + private static int CountGitAttributeTokens(ReadOnlySpan value) + { + var count = 0; + while (!value.IsEmpty) + { + var separator = value.IndexOfAny(' ', '\t'); + var token = separator < 0 ? value : value[..separator]; + if (!token.Trim().IsEmpty) + count++; + + if (separator < 0) + break; + value = value[(separator + 1)..]; + } + + return count; + } + private static bool TryAppendGitAttributesEscape( ReadOnlySpan value, ref int index, diff --git a/tests/CodeIndex.Tests/PerformanceTests.cs b/tests/CodeIndex.Tests/PerformanceTests.cs index cf0c1c1a62..4f0af20c17 100644 --- a/tests/CodeIndex.Tests/PerformanceTests.cs +++ b/tests/CodeIndex.Tests/PerformanceTests.cs @@ -443,6 +443,31 @@ public void CppHeaderDetection_LargeSample_DoesNotMaterializeLineArrays() $"C/C++ header detection allocated {allocatedBytes:N0} bytes"); } +#if NET8_0 + [Fact] +#else + [Fact(Skip = PracticalBudgetTestTarget.SecondaryTargetSkipReason)] +#endif + public void DelimitedSpanWalking_DenseExtractorLists_DoesNotAllocate() + { + var content = string.Join( + ',', + Enumerable.Range(0, 8_192) + .Select(index => $" value_{index} ")); + var expectedLength = Enumerable.Range(0, 8_192) + .Sum(index => $"value_{index}".Length); + _ = MeasureDelimitedEntries(content); + + (int Count, int TotalLength) result = default; + var allocatedBytes = MeasureAllocatedBytes( + () => result = MeasureDelimitedEntries(content)); + + Assert.Equal((8_192, expectedLength), result); + Assert.True( + allocatedBytes < 1_024, + $"Delimited span walking allocated {allocatedBytes:N0} bytes"); + } + #if NET8_0 [Fact] #else @@ -653,6 +678,23 @@ private static long MeasureAllocatedBytes(Action action) return GC.GetAllocatedBytesForCurrentThread() - before; } + private static (int Count, int TotalLength) MeasureDelimitedEntries(string content) + { + var count = 0; + var totalLength = 0; + foreach (var entry in new DelimitedSpanEnumerable( + content.AsSpan(), + ',', + trimEntries: true, + removeEmptyEntries: true)) + { + count++; + totalLength += entry.Length; + } + + return (count, totalLength); + } + private static TimeSpan MeasureElapsed(Action action) { var sw = Stopwatch.StartNew(); From 55ec874389149c8cd037d6cbd21f72c60cd23eda Mon Sep 17 00:00:00 2001 From: Widthdom Date: Sun, 26 Jul 2026 17:12:29 +0900 Subject: [PATCH 04/30] Avoid functional span predicate closures --- DEVELOPER_GUIDE.md | 7 +++ TESTING_GUIDE.md | 2 + .../+functional-span-membership.fixed.md | 23 ++++++++++ ...tionalLanguageReferenceExtractor.Erlang.cs | 4 +- ...ctionalLanguageReferenceExtractor.Ocaml.cs | 10 ++--- ...nctionalLanguageReferenceExtractor.Raku.cs | 2 +- .../FunctionalLanguageReferenceExtractor.cs | 43 +++++++++++++++++++ tests/CodeIndex.Tests/PerformanceTests.cs | 32 ++++++++++++++ 8 files changed, 115 insertions(+), 8 deletions(-) create mode 100644 changelog.d/unreleased/+functional-span-membership.fixed.md diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index 2f88605cae..b09efdbb46 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -511,6 +511,10 @@ trimming, quote, and first-separator semantics when replacing it. `DelimitedSpanEnumerable` is the shared allocation-free walker for single delimiters; repository metadata, application manifests, VHDL declarations and package paths, and CUDA parameter headers use it instead of split arrays. +Likewise, exclusion-range checks inside dense match loops must use indexed +helpers rather than capturing LINQ predicates. Erlang, OCaml, and Raku share +`ContainsFunctionalSpan` / `OverlapsFunctionalSpan` for remote, qualified, +quoted-atom, and type-reference suppression. When structural masking turns a source line into whitespace, do not materialize a trimmed copy merely to discover that the line has no references. Preserve @@ -3510,6 +3514,9 @@ import、dependency、path segment、declaration item ごとに array と substr single delimiter には allocation-free な共通 walker `DelimitedSpanEnumerable` を使う。 repository metadata、application manifest、VHDL declaration / package path、CUDA parameter header は split array を作らずこの walker で処理する。 +同様に、dense match loop 内の exclusion-range 判定で capturing LINQ predicate を使っては +ならない。Erlang、OCaml、Raku は remote / qualified / quoted-atom / type-reference の +抑制に `ContainsFunctionalSpan` / `OverlapsFunctionalSpan` を共有する。 構造マスクによって source line が空白だけになった場合、reference がないことを確認するため だけに trim 済み copy を実体化してはならない。documentation handling と、original line を diff --git a/TESTING_GUIDE.md b/TESTING_GUIDE.md index 1810bcdbce..8c017c731e 100644 --- a/TESTING_GUIDE.md +++ b/TESTING_GUIDE.md @@ -579,6 +579,7 @@ Use `docs/test-doc-maintenance-plan.md` before moving oversized suites or adding `ReferenceExtraction_MaskedMultilinePayloads_StayWithinAllocationBudget` keeps C# raw strings, Java text blocks, and TypeScript template literals from materializing trimmed reference contexts after structural masking has made a line empty. `CppHeaderDetection_LargeSample_DoesNotMaterializeLineArrays` keeps bounded C / C++ header-disambiguation samples on span-based line walks instead of allocating a string and array for every sampled line. `DelimitedSpanWalking_DenseExtractorLists_DoesNotAllocate` locks the shared single-delimiter walker to allocation-free trim/remove-empty semantics used by repository metadata, application manifests, VHDL, and CUDA extraction. + `FunctionalSpanMembership_RepeatedCallFiltering_DoesNotAllocate` prevents per-match capturing-predicate allocations from returning to Erlang, OCAML, and Raku exclusion-range checks. `ReferenceExtraction_CSharpNoAliasDenseReferences_StaysWithinAllocationBudget` keeps 12,000 no-alias calls from paying for alias dedupe, `CSharpAliasCompaction_DenseDuplicates_StaysWithinAllocationBudget` guards pre-sized stable compaction after alias rewrites, and `MutualRecursion_DenseRepeatedQualifiedNames_StaysWithinAllocationBudget` covers repeated qualified C#- and Python-style cycle names without per-edge normalization strings. `ReusableStatSnapshot_OnePassMaterialization_StaysWithinAllocationBudget` warms a 1,024-row typed snapshot with its exact capacity, then prevents a second candidate collection or redundant path values from returning to the default `net8.0` path. `ReferenceBatchTransactions_RepositoryScaleAtomicFileScopeEliminatesControlledSqlScopes` models 321,352 references distributed over 856 files in the repository snapshot's five/six-batch shape and fixes the control-SQL contract at 5,009 public batch transaction scopes versus zero explicit atomic-file batch scopes; keep it deterministic and allocation-light instead of inserting all modeled rows or asserting elapsed time. For reference-line window performance audits, measure identical prebuilt reference rows against one-batch and 32-batch limits in alternating order, report both elapsed time and `GC.GetAllocatedBytesForCurrentThread`, and remove the timing harness after recording the result; end-to-end `--memory-trace` rebuilds remain corroborating evidence because extraction, graph finalization, and OS page-cache variance can dominate the persistence delta. @@ -1431,6 +1432,7 @@ dotnet test --filter "FullyQualifiedName~GitHelperTests" `ReferenceExtraction_MaskedMultilinePayloads_StayWithinAllocationBudget` は、構造マスク後に空行となった C# raw string、Java text block、TypeScript template literal から trim 済み reference context を実体化しないことを固定します。 `CppHeaderDetection_LargeSample_DoesNotMaterializeLineArrays` は、bounded な C / C++ header 判定 sample を span ベースで行走査し、sampled line ごとの string と array を割り当てないことを固定します。 `DelimitedSpanWalking_DenseExtractorLists_DoesNotAllocate` は、repository metadata、application manifest、VHDL、CUDA extraction が共有する single-delimiter walker の trim / remove-empty semantics を allocation-free に固定します。 + `FunctionalSpanMembership_RepeatedCallFiltering_DoesNotAllocate` は、Erlang、OCAML、Raku の exclusion-range 判定へ match ごとの capturing-predicate allocation が戻らないことを固定します。 `ReferenceExtraction_CSharpNoAliasDenseReferences_StaysWithinAllocationBudget` は alias のない 12,000 call が alias dedupe のコストを負わないこと、`CSharpAliasCompaction_DenseDuplicates_StaysWithinAllocationBudget` は alias rewrite 後の事前 capacity 付き stable compaction、`MutualRecursion_DenseRepeatedQualifiedNames_StaysWithinAllocationBudget` は C# / Python 形式の qualified cycle name が edge ごとの正規化文字列を作らないことを保証します。 `ReusableStatSnapshot_OnePassMaterialization_StaysWithinAllocationBudget` は exact capacity を渡した1,024行の typed snapshot を warm-up し、2つ目の候補 collection や重複 path value が通常の `net8.0` 経路へ戻らないようにします。 `ReferenceBatchTransactions_RepositoryScaleAtomicFileScopeEliminatesControlledSqlScopes` は321,352 refsを自己snapshotの5/6 batch形状で856 filesへ分配し、制御SQL契約をpublic batch transaction scope 5,009回対explicit atomic-file scope 0回に固定します。全model rowを挿入したり経過時間をassertしたりせず、deterministicでallocation-lightなまま維持してください。reference-line windowの性能監査では、同一の事前構築済みreference rowを1-batch上限と32-batch上限で交互に測り、経過時間と`GC.GetAllocatedBytesForCurrentThread`の両方を報告して、結果記録後にtiming harnessを削除します。end-to-endの`--memory-trace` rebuildは、extraction・graph finalize・OS page cacheの変動が永続化差を支配し得るため、補助証拠として扱ってください。 diff --git a/changelog.d/unreleased/+functional-span-membership.fixed.md b/changelog.d/unreleased/+functional-span-membership.fixed.md new file mode 100644 index 0000000000..b7164ac8f4 --- /dev/null +++ b/changelog.d/unreleased/+functional-span-membership.fixed.md @@ -0,0 +1,23 @@ +--- +category: fixed +affected: + - src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.cs + - src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Erlang.cs + - src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Ocaml.cs + - src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Raku.cs + - tests/CodeIndex.Tests/PerformanceTests.cs + - DEVELOPER_GUIDE.md + - TESTING_GUIDE.md +--- + +## English + +- **Removed per-match closure allocations from functional-language references** — + Erlang, OCAML, and Raku now use shared indexed span-membership checks when + suppressing remote, qualified, quoted-atom, and type-reference matches. + +## 日本語 + +- **functional-language reference の match ごとの closure allocation を解消しました** — + Erlang、OCAML、Raku は remote、qualified、quoted-atom、type-reference match の抑制時に、 + 共通の indexed span-membership check を使います。 diff --git a/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Erlang.cs b/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Erlang.cs index 36f1b12198..b69d743972 100644 --- a/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Erlang.cs +++ b/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Erlang.cs @@ -42,7 +42,7 @@ private static void EmitErlangReferences( var definitionMatch = ErlangFunctionDefinitionRegex.Match(line); foreach (Match match in ErlangLocalCallRegex.Matches(line)) { - if (remoteCallSpans.Any(span => match.Index >= span.Start && match.Index < span.End)) + if (ContainsFunctionalSpan(remoteCallSpans, match.Index)) continue; if (IsInsideQuotedAtom(match.Groups["name"].Index)) continue; @@ -67,7 +67,7 @@ void AddFunctionalMatchReference(Match match, string kind) } bool IsInsideQuotedAtom(int index) - => quotedAtomSpans.Any(span => index > span.Start && index < span.End); + => ContainsFunctionalSpanInterior(quotedAtomSpans, index); } private static List<(int Start, int End)> GetErlangQuotedAtomSpans(string line) diff --git a/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Ocaml.cs b/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Ocaml.cs index a011afc47d..09db3a92a3 100644 --- a/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Ocaml.cs +++ b/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Ocaml.cs @@ -66,7 +66,10 @@ private static void EmitOcamlReferences( var qualifiedCallSpans = new List<(int Start, int End)>(typeReferenceSpans); foreach (Match match in OcamlQualifiedCallRegex.Matches(line)) { - if (qualifiedCallSpans.Any(span => RangesOverlap(span.Start, span.End, match.Index, match.Index + match.Length))) + if (OverlapsFunctionalSpan( + qualifiedCallSpans, + match.Index, + match.Index + match.Length)) continue; qualifiedCallSpans.Add((match.Index, match.Index + match.Length)); AddFunctionalReference(references, seen, fileId, match.Groups["module"], "reference", context, lineNumber, container, "ocaml"); @@ -76,7 +79,7 @@ private static void EmitOcamlReferences( var skippedDefinition = false; foreach (Match match in OcamlBareCallRegex.Matches(line)) { - if (qualifiedCallSpans.Any(span => match.Index >= span.Start && match.Index < span.End)) + if (ContainsFunctionalSpan(qualifiedCallSpans, match.Index)) continue; var name = match.Groups["name"].Value; @@ -120,9 +123,6 @@ void AddOcamlTypeReference(Group group) "ocaml"); } } - - static bool RangesOverlap(int leftStart, int leftEnd, int rightStart, int rightEnd) - => leftStart < rightEnd && rightStart < leftEnd; } } diff --git a/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Raku.cs b/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Raku.cs index a6c808fa5f..20f453fd49 100644 --- a/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Raku.cs +++ b/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Raku.cs @@ -52,7 +52,7 @@ private static void EmitRakuReferences( var skippedDefinition = false; foreach (Match match in RakuBareCallRegex.Matches(line)) { - if (qualifiedCallSpans.Any(span => match.Index >= span.Start && match.Index < span.End)) + if (ContainsFunctionalSpan(qualifiedCallSpans, match.Index)) continue; var name = match.Groups["name"].Value; diff --git a/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.cs b/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.cs index 4f7de8e5d6..f10b4ddbb1 100644 --- a/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.cs +++ b/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.cs @@ -138,6 +138,49 @@ private sealed class FunctionalReferenceState internal SymbolRecord? ActiveCallable; } + internal static bool ContainsFunctionalSpan( + IReadOnlyList<(int Start, int End)> spans, + int index) + { + for (var spanIndex = 0; spanIndex < spans.Count; spanIndex++) + { + var span = spans[spanIndex]; + if (index >= span.Start && index < span.End) + return true; + } + + return false; + } + + private static bool ContainsFunctionalSpanInterior( + IReadOnlyList<(int Start, int End)> spans, + int index) + { + for (var spanIndex = 0; spanIndex < spans.Count; spanIndex++) + { + var span = spans[spanIndex]; + if (index > span.Start && index < span.End) + return true; + } + + return false; + } + + internal static bool OverlapsFunctionalSpan( + IReadOnlyList<(int Start, int End)> spans, + int start, + int end) + { + for (var spanIndex = 0; spanIndex < spans.Count; spanIndex++) + { + var span = spans[spanIndex]; + if (span.Start < end && start < span.End) + return true; + } + + return false; + } + private static List ExtractFunctionalLanguageReferences(ReferenceExtractionContext request) { if (!TryPrepareReferenceLines( diff --git a/tests/CodeIndex.Tests/PerformanceTests.cs b/tests/CodeIndex.Tests/PerformanceTests.cs index 4f0af20c17..221c43a22e 100644 --- a/tests/CodeIndex.Tests/PerformanceTests.cs +++ b/tests/CodeIndex.Tests/PerformanceTests.cs @@ -468,6 +468,38 @@ public void DelimitedSpanWalking_DenseExtractorLists_DoesNotAllocate() $"Delimited span walking allocated {allocatedBytes:N0} bytes"); } +#if NET8_0 + [Fact] +#else + [Fact(Skip = PracticalBudgetTestTarget.SecondaryTargetSkipReason)] +#endif + public void FunctionalSpanMembership_RepeatedCallFiltering_DoesNotAllocate() + { + var spans = Enumerable.Range(0, 128) + .Select(index => (Start: index * 8, End: index * 8 + 4)) + .ToArray(); + var foundCount = 0; + _ = ReferenceExtractor.ContainsFunctionalSpan(spans, 16); + + var allocatedBytes = MeasureAllocatedBytes(() => + { + for (var index = 0; index < 10_000; index++) + { + if (ReferenceExtractor.ContainsFunctionalSpan( + spans, + index % (spans[^1].End + 1))) + { + foundCount++; + } + } + }); + + Assert.True(foundCount > 0); + Assert.True( + allocatedBytes < 1_024, + $"Functional span membership allocated {allocatedBytes:N0} bytes"); + } + #if NET8_0 [Fact] #else From 35d8fc08efaa2f5ee6ae55f1f10a132a8d843119 Mon Sep 17 00:00:00 2001 From: Widthdom Date: Sun, 26 Jul 2026 17:16:52 +0900 Subject: [PATCH 05/30] Avoid hardware scope predicate closures --- DEVELOPER_GUIDE.md | 7 +++ TESTING_GUIDE.md | 2 + .../+hardware-scope-membership.fixed.md | 21 +++++++ .../Languages/HdlReferenceExtractor.cs | 15 ++++- .../Languages/ShaderReferenceExtractor.cs | 50 +++++++++++++--- tests/CodeIndex.Tests/PerformanceTests.cs | 59 +++++++++++++++++++ 6 files changed, 146 insertions(+), 8 deletions(-) create mode 100644 changelog.d/unreleased/+hardware-scope-membership.fixed.md diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index b09efdbb46..a832f436ee 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -515,6 +515,10 @@ Likewise, exclusion-range checks inside dense match loops must use indexed helpers rather than capturing LINQ predicates. Erlang, OCaml, and Raku share `ContainsFunctionalSpan` / `OverlapsFunctionalSpan` for remote, qualified, quoted-atom, and type-reference suppression. +The same rule applies to hardware languages: Verilog / SystemVerilog / VHDL +shadow scopes and CUDA / GLSL / HLSL / Metal / WGSL binding and resource scopes +use direct indexed loops so every identifier does not allocate a predicate +closure. When structural masking turns a source line into whitespace, do not materialize a trimmed copy merely to discover that the line has no references. Preserve @@ -3517,6 +3521,9 @@ header は split array を作らずこの walker で処理する。 同様に、dense match loop 内の exclusion-range 判定で capturing LINQ predicate を使っては ならない。Erlang、OCaml、Raku は remote / qualified / quoted-atom / type-reference の 抑制に `ContainsFunctionalSpan` / `OverlapsFunctionalSpan` を共有する。 +hardware language も同じ規則に従う。Verilog / SystemVerilog / VHDL の shadow scope と、 +CUDA / GLSL / HLSL / Metal / WGSL の binding / resource scope は direct indexed loop を +使い、identifier ごとの predicate closure を作らない。 構造マスクによって source line が空白だけになった場合、reference がないことを確認するため だけに trim 済み copy を実体化してはならない。documentation handling と、original line を diff --git a/TESTING_GUIDE.md b/TESTING_GUIDE.md index 8c017c731e..383b080e38 100644 --- a/TESTING_GUIDE.md +++ b/TESTING_GUIDE.md @@ -580,6 +580,7 @@ Use `docs/test-doc-maintenance-plan.md` before moving oversized suites or adding `CppHeaderDetection_LargeSample_DoesNotMaterializeLineArrays` keeps bounded C / C++ header-disambiguation samples on span-based line walks instead of allocating a string and array for every sampled line. `DelimitedSpanWalking_DenseExtractorLists_DoesNotAllocate` locks the shared single-delimiter walker to allocation-free trim/remove-empty semantics used by repository metadata, application manifests, VHDL, and CUDA extraction. `FunctionalSpanMembership_RepeatedCallFiltering_DoesNotAllocate` prevents per-match capturing-predicate allocations from returning to Erlang, OCAML, and Raku exclusion-range checks. + `HardwareScopeMembership_RepeatedIdentifierFiltering_DoesNotAllocate` prevents per-identifier predicate closures from returning to shader binding/resource scope checks shared by CUDA, GLSL, HLSL, Metal, and WGSL; the corresponding HDL extraction regression covers shadowing semantics for Verilog, SystemVerilog, and VHDL. `ReferenceExtraction_CSharpNoAliasDenseReferences_StaysWithinAllocationBudget` keeps 12,000 no-alias calls from paying for alias dedupe, `CSharpAliasCompaction_DenseDuplicates_StaysWithinAllocationBudget` guards pre-sized stable compaction after alias rewrites, and `MutualRecursion_DenseRepeatedQualifiedNames_StaysWithinAllocationBudget` covers repeated qualified C#- and Python-style cycle names without per-edge normalization strings. `ReusableStatSnapshot_OnePassMaterialization_StaysWithinAllocationBudget` warms a 1,024-row typed snapshot with its exact capacity, then prevents a second candidate collection or redundant path values from returning to the default `net8.0` path. `ReferenceBatchTransactions_RepositoryScaleAtomicFileScopeEliminatesControlledSqlScopes` models 321,352 references distributed over 856 files in the repository snapshot's five/six-batch shape and fixes the control-SQL contract at 5,009 public batch transaction scopes versus zero explicit atomic-file batch scopes; keep it deterministic and allocation-light instead of inserting all modeled rows or asserting elapsed time. For reference-line window performance audits, measure identical prebuilt reference rows against one-batch and 32-batch limits in alternating order, report both elapsed time and `GC.GetAllocatedBytesForCurrentThread`, and remove the timing harness after recording the result; end-to-end `--memory-trace` rebuilds remain corroborating evidence because extraction, graph finalization, and OS page-cache variance can dominate the persistence delta. @@ -1433,6 +1434,7 @@ dotnet test --filter "FullyQualifiedName~GitHelperTests" `CppHeaderDetection_LargeSample_DoesNotMaterializeLineArrays` は、bounded な C / C++ header 判定 sample を span ベースで行走査し、sampled line ごとの string と array を割り当てないことを固定します。 `DelimitedSpanWalking_DenseExtractorLists_DoesNotAllocate` は、repository metadata、application manifest、VHDL、CUDA extraction が共有する single-delimiter walker の trim / remove-empty semantics を allocation-free に固定します。 `FunctionalSpanMembership_RepeatedCallFiltering_DoesNotAllocate` は、Erlang、OCAML、Raku の exclusion-range 判定へ match ごとの capturing-predicate allocation が戻らないことを固定します。 + `HardwareScopeMembership_RepeatedIdentifierFiltering_DoesNotAllocate` は、CUDA、GLSL、HLSL、Metal、WGSL が共有する shader binding / resource scope 判定へ identifier ごとの predicate closure が戻らないことを固定し、対応する HDL extraction regression が Verilog、SystemVerilog、VHDL の shadowing semantics をカバーします。 `ReferenceExtraction_CSharpNoAliasDenseReferences_StaysWithinAllocationBudget` は alias のない 12,000 call が alias dedupe のコストを負わないこと、`CSharpAliasCompaction_DenseDuplicates_StaysWithinAllocationBudget` は alias rewrite 後の事前 capacity 付き stable compaction、`MutualRecursion_DenseRepeatedQualifiedNames_StaysWithinAllocationBudget` は C# / Python 形式の qualified cycle name が edge ごとの正規化文字列を作らないことを保証します。 `ReusableStatSnapshot_OnePassMaterialization_StaysWithinAllocationBudget` は exact capacity を渡した1,024行の typed snapshot を warm-up し、2つ目の候補 collection や重複 path value が通常の `net8.0` 経路へ戻らないようにします。 `ReferenceBatchTransactions_RepositoryScaleAtomicFileScopeEliminatesControlledSqlScopes` は321,352 refsを自己snapshotの5/6 batch形状で856 filesへ分配し、制御SQL契約をpublic batch transaction scope 5,009回対explicit atomic-file scope 0回に固定します。全model rowを挿入したり経過時間をassertしたりせず、deterministicでallocation-lightなまま維持してください。reference-line windowの性能監査では、同一の事前構築済みreference rowを1-batch上限と32-batch上限で交互に測り、経過時間と`GC.GetAllocatedBytesForCurrentThread`の両方を報告して、結果記録後にtiming harnessを削除します。end-to-endの`--memory-trace` rebuildは、extraction・graph finalize・OS page cacheの変動が永続化差を支配し得るため、補助証拠として扱ってください。 diff --git a/changelog.d/unreleased/+hardware-scope-membership.fixed.md b/changelog.d/unreleased/+hardware-scope-membership.fixed.md new file mode 100644 index 0000000000..e6c742085a --- /dev/null +++ b/changelog.d/unreleased/+hardware-scope-membership.fixed.md @@ -0,0 +1,21 @@ +--- +category: fixed +affected: + - src/CodeIndex/Indexer/References/Languages/HdlReferenceExtractor.cs + - src/CodeIndex/Indexer/References/Languages/ShaderReferenceExtractor.cs + - tests/CodeIndex.Tests/PerformanceTests.cs + - DEVELOPER_GUIDE.md + - TESTING_GUIDE.md +--- + +## English + +- **Removed per-identifier closure allocations from hardware-language scopes** — + Verilog, SystemVerilog, and VHDL shadow checks plus CUDA, GLSL, HLSL, Metal, + and WGSL binding/resource checks now use direct indexed membership loops. + +## 日本語 + +- **hardware-language scope の identifier ごとの closure allocation を解消しました** — + Verilog、SystemVerilog、VHDL の shadow 判定と、CUDA、GLSL、HLSL、Metal、WGSL + の binding / resource 判定は direct indexed membership loop を使います。 diff --git a/src/CodeIndex/Indexer/References/Languages/HdlReferenceExtractor.cs b/src/CodeIndex/Indexer/References/Languages/HdlReferenceExtractor.cs index 96d11820b5..7bbf7e3443 100644 --- a/src/CodeIndex/Indexer/References/Languages/HdlReferenceExtractor.cs +++ b/src/CodeIndex/Indexer/References/Languages/HdlReferenceExtractor.cs @@ -687,7 +687,7 @@ private static void EmitKnownHdlReferences( || definitionsByLine.TryGetValue(lineNumber, out var definitions) && definitions.Contains(match.Value) || declaredNames?.Contains(match.Value) == true - || scopes.Any(scope => scope.ShadowedNames.Contains(match.Value)) + || IsHdlNameShadowed(scopes, match.Value) || knownSymbol.LocalDesignUnitIds is { Count: > 0 } localDesignUnitIds && !localDesignUnitIds.Contains(currentDesignUnitId)) { @@ -716,6 +716,19 @@ private static void EmitKnownHdlReferences( } } + private static bool IsHdlNameShadowed( + IReadOnlyList scopes, + string name) + { + for (var scopeIndex = 0; scopeIndex < scopes.Count; scopeIndex++) + { + if (scopes[scopeIndex].ShadowedNames.Contains(name)) + return true; + } + + return false; + } + private static bool IsFollowedByOpenParenthesis(string line, int index) { while (index < line.Length && char.IsWhiteSpace(line[index])) diff --git a/src/CodeIndex/Indexer/References/Languages/ShaderReferenceExtractor.cs b/src/CodeIndex/Indexer/References/Languages/ShaderReferenceExtractor.cs index 2cdf80a9d4..3e1c860c9c 100644 --- a/src/CodeIndex/Indexer/References/Languages/ShaderReferenceExtractor.cs +++ b/src/CodeIndex/Indexer/References/Languages/ShaderReferenceExtractor.cs @@ -672,10 +672,25 @@ private static void AddBindingSite( state.BindingsByLine[lineNumber] = bindings; } - if (!bindings.Any(binding => binding.Name == name && binding.Column == column)) + if (!ContainsBindingSite(bindings, name, column)) bindings.Add(new BindingSite(name, column)); } + internal static bool ContainsBindingSite( + IReadOnlyList bindings, + string name, + int column) + { + for (var bindingIndex = 0; bindingIndex < bindings.Count; bindingIndex++) + { + var binding = bindings[bindingIndex]; + if (binding.Name == name && binding.Column == column) + return true; + } + + return false; + } + private static void TrackResourceDefinition(State state, string name, int lineNumber) { if (!state.TryTrackName(state.ResourceNames, name)) @@ -823,12 +838,11 @@ private static void EmitTrackedNameReferences( var isGlobalResource = state.GlobalResourceNames.Contains(name); var isScopedResource = state.ScopedResourcesByName.TryGetValue(name, out var scopes) - && scopes.Any(scope => - scope.ContainerName == container.Name - && (lineNumber > scope.HeaderEndLine - || (lineNumber == scope.HeaderEndLine - && match.Index >= scope.FirstBodyColumn)) - && lineNumber <= scope.BodyEndLine); + && ContainsActiveScopedResource( + scopes, + container.Name, + lineNumber, + match.Index); if (!isGlobalResource && !isScopedResource) continue; @@ -845,4 +859,26 @@ private static void EmitTrackedNameReferences( state.Language); } } + + internal static bool ContainsActiveScopedResource( + IReadOnlyList scopes, + string containerName, + int lineNumber, + int column) + { + for (var scopeIndex = 0; scopeIndex < scopes.Count; scopeIndex++) + { + var scope = scopes[scopeIndex]; + if (scope.ContainerName == containerName + && (lineNumber > scope.HeaderEndLine + || (lineNumber == scope.HeaderEndLine + && column >= scope.FirstBodyColumn)) + && lineNumber <= scope.BodyEndLine) + { + return true; + } + } + + return false; + } } diff --git a/tests/CodeIndex.Tests/PerformanceTests.cs b/tests/CodeIndex.Tests/PerformanceTests.cs index 221c43a22e..1bb14d0c53 100644 --- a/tests/CodeIndex.Tests/PerformanceTests.cs +++ b/tests/CodeIndex.Tests/PerformanceTests.cs @@ -500,6 +500,65 @@ public void FunctionalSpanMembership_RepeatedCallFiltering_DoesNotAllocate() $"Functional span membership allocated {allocatedBytes:N0} bytes"); } +#if NET8_0 + [Fact] +#else + [Fact(Skip = PracticalBudgetTestTarget.SecondaryTargetSkipReason)] +#endif + public void HardwareScopeMembership_RepeatedIdentifierFiltering_DoesNotAllocate() + { + var bindings = Enumerable.Range(0, 128) + .Select(index => new ShaderReferenceExtractor.BindingSite( + $"resource_{index}", + index * 4)) + .ToArray(); + var scopes = Enumerable.Range(0, 128) + .Select(index => new ShaderReferenceExtractor.ScopedResource( + $"kernel_{index}", + HeaderEndLine: index, + BodyEndLine: index + 16, + FirstBodyColumn: 8)) + .ToArray(); + var missingNames = Enumerable.Range(0, 8) + .Select(index => $"missing_{index}") + .ToArray(); + var matchCount = 0; + _ = ShaderReferenceExtractor.ContainsBindingSite(bindings, "resource_64", 256); + _ = ShaderReferenceExtractor.ContainsActiveScopedResource( + scopes, + "kernel_64", + 70, + 12); + + var allocatedBytes = MeasureAllocatedBytes(() => + { + for (var index = 0; index < 10_000; index++) + { + if (ShaderReferenceExtractor.ContainsBindingSite( + bindings, + missingNames[index % missingNames.Length], + index)) + { + matchCount++; + } + + if (ShaderReferenceExtractor.ContainsActiveScopedResource( + scopes, + "kernel_64", + 70, + 12)) + { + matchCount++; + } + } + }); + + Assert.Equal(10_000, matchCount); + Assert.True( + allocatedBytes < 1_024, + $"Hardware scope membership allocated {allocatedBytes:N0} bytes"); + } + #if NET8_0 [Fact] #else From 5c04709f645891277fb299808964df9c54fda8ea Mon Sep 17 00:00:00 2001 From: Widthdom Date: Sun, 26 Jul 2026 17:19:26 +0900 Subject: [PATCH 06/30] Stream metadata character and ancestor checks --- DEVELOPER_GUIDE.md | 7 +++++ TESTING_GUIDE.md | 2 ++ .../+metadata-character-scans.fixed.md | 24 +++++++++++++++ .../RepositoryMetadataReferenceExtractor.cs | 29 ++++++++++-------- src/CodeIndex/Indexer/SpanCharacterSearch.cs | 26 ++++++++++++++++ .../SymbolExtractor.RepositoryMetadata.cs | 8 +++-- tests/CodeIndex.Tests/PerformanceTests.cs | 30 +++++++++++++++++++ 7 files changed, 111 insertions(+), 15 deletions(-) create mode 100644 changelog.d/unreleased/+metadata-character-scans.fixed.md create mode 100644 src/CodeIndex/Indexer/SpanCharacterSearch.cs diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index a832f436ee..5546ad5a62 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -519,6 +519,10 @@ The same rule applies to hardware languages: Verilog / SystemVerilog / VHDL shadow scopes and CUDA / GLSL / HLSL / Metal / WGSL binding and resource scopes use direct indexed loops so every identifier does not allocate a predicate closure. +Repository metadata character validation uses `SpanCharacterSearch` instead of +predicate-based enumeration, and application-manifest dependency ownership is +tracked as XML depth rather than rescanning an ancestor stack for every +`assemblyIdentity`. When structural masking turns a source line into whitespace, do not materialize a trimmed copy merely to discover that the line has no references. Preserve @@ -3524,6 +3528,9 @@ header は split array を作らずこの walker で処理する。 hardware language も同じ規則に従う。Verilog / SystemVerilog / VHDL の shadow scope と、 CUDA / GLSL / HLSL / Metal / WGSL の binding / resource scope は direct indexed loop を 使い、identifier ごとの predicate closure を作らない。 +repository metadata の character validation は predicate-based enumeration ではなく +`SpanCharacterSearch` を使い、application manifest の dependency ownership は +`assemblyIdentity` ごとの ancestor stack 再走査ではなく XML depth で追跡する。 構造マスクによって source line が空白だけになった場合、reference がないことを確認するため だけに trim 済み copy を実体化してはならない。documentation handling と、original line を diff --git a/TESTING_GUIDE.md b/TESTING_GUIDE.md index 383b080e38..b75d8b80dc 100644 --- a/TESTING_GUIDE.md +++ b/TESTING_GUIDE.md @@ -581,6 +581,7 @@ Use `docs/test-doc-maintenance-plan.md` before moving oversized suites or adding `DelimitedSpanWalking_DenseExtractorLists_DoesNotAllocate` locks the shared single-delimiter walker to allocation-free trim/remove-empty semantics used by repository metadata, application manifests, VHDL, and CUDA extraction. `FunctionalSpanMembership_RepeatedCallFiltering_DoesNotAllocate` prevents per-match capturing-predicate allocations from returning to Erlang, OCAML, and Raku exclusion-range checks. `HardwareScopeMembership_RepeatedIdentifierFiltering_DoesNotAllocate` prevents per-identifier predicate closures from returning to shader binding/resource scope checks shared by CUDA, GLSL, HLSL, Metal, and WGSL; the corresponding HDL extraction regression covers shadowing semantics for Verilog, SystemVerilog, and VHDL. + `SpanCharacterSearch_RepeatedLongMetadataCandidates_DoesNotAllocate` keeps control/whitespace validation allocation-free for long repository-metadata candidates; application-manifest regressions separately preserve dependency ancestry and local probing-path behavior. `ReferenceExtraction_CSharpNoAliasDenseReferences_StaysWithinAllocationBudget` keeps 12,000 no-alias calls from paying for alias dedupe, `CSharpAliasCompaction_DenseDuplicates_StaysWithinAllocationBudget` guards pre-sized stable compaction after alias rewrites, and `MutualRecursion_DenseRepeatedQualifiedNames_StaysWithinAllocationBudget` covers repeated qualified C#- and Python-style cycle names without per-edge normalization strings. `ReusableStatSnapshot_OnePassMaterialization_StaysWithinAllocationBudget` warms a 1,024-row typed snapshot with its exact capacity, then prevents a second candidate collection or redundant path values from returning to the default `net8.0` path. `ReferenceBatchTransactions_RepositoryScaleAtomicFileScopeEliminatesControlledSqlScopes` models 321,352 references distributed over 856 files in the repository snapshot's five/six-batch shape and fixes the control-SQL contract at 5,009 public batch transaction scopes versus zero explicit atomic-file batch scopes; keep it deterministic and allocation-light instead of inserting all modeled rows or asserting elapsed time. For reference-line window performance audits, measure identical prebuilt reference rows against one-batch and 32-batch limits in alternating order, report both elapsed time and `GC.GetAllocatedBytesForCurrentThread`, and remove the timing harness after recording the result; end-to-end `--memory-trace` rebuilds remain corroborating evidence because extraction, graph finalization, and OS page-cache variance can dominate the persistence delta. @@ -1435,6 +1436,7 @@ dotnet test --filter "FullyQualifiedName~GitHelperTests" `DelimitedSpanWalking_DenseExtractorLists_DoesNotAllocate` は、repository metadata、application manifest、VHDL、CUDA extraction が共有する single-delimiter walker の trim / remove-empty semantics を allocation-free に固定します。 `FunctionalSpanMembership_RepeatedCallFiltering_DoesNotAllocate` は、Erlang、OCAML、Raku の exclusion-range 判定へ match ごとの capturing-predicate allocation が戻らないことを固定します。 `HardwareScopeMembership_RepeatedIdentifierFiltering_DoesNotAllocate` は、CUDA、GLSL、HLSL、Metal、WGSL が共有する shader binding / resource scope 判定へ identifier ごとの predicate closure が戻らないことを固定し、対応する HDL extraction regression が Verilog、SystemVerilog、VHDL の shadowing semantics をカバーします。 + `SpanCharacterSearch_RepeatedLongMetadataCandidates_DoesNotAllocate` は、長い repository-metadata candidate の control / whitespace validation を allocation-free に固定し、application-manifest regression が dependency ancestry と local probing-path の意味論を別途維持します。 `ReferenceExtraction_CSharpNoAliasDenseReferences_StaysWithinAllocationBudget` は alias のない 12,000 call が alias dedupe のコストを負わないこと、`CSharpAliasCompaction_DenseDuplicates_StaysWithinAllocationBudget` は alias rewrite 後の事前 capacity 付き stable compaction、`MutualRecursion_DenseRepeatedQualifiedNames_StaysWithinAllocationBudget` は C# / Python 形式の qualified cycle name が edge ごとの正規化文字列を作らないことを保証します。 `ReusableStatSnapshot_OnePassMaterialization_StaysWithinAllocationBudget` は exact capacity を渡した1,024行の typed snapshot を warm-up し、2つ目の候補 collection や重複 path value が通常の `net8.0` 経路へ戻らないようにします。 `ReferenceBatchTransactions_RepositoryScaleAtomicFileScopeEliminatesControlledSqlScopes` は321,352 refsを自己snapshotの5/6 batch形状で856 filesへ分配し、制御SQL契約をpublic batch transaction scope 5,009回対explicit atomic-file scope 0回に固定します。全model rowを挿入したり経過時間をassertしたりせず、deterministicでallocation-lightなまま維持してください。reference-line windowの性能監査では、同一の事前構築済みreference rowを1-batch上限と32-batch上限で交互に測り、経過時間と`GC.GetAllocatedBytesForCurrentThread`の両方を報告して、結果記録後にtiming harnessを削除します。end-to-endの`--memory-trace` rebuildは、extraction・graph finalize・OS page cacheの変動が永続化差を支配し得るため、補助証拠として扱ってください。 diff --git a/changelog.d/unreleased/+metadata-character-scans.fixed.md b/changelog.d/unreleased/+metadata-character-scans.fixed.md new file mode 100644 index 0000000000..066336c5f0 --- /dev/null +++ b/changelog.d/unreleased/+metadata-character-scans.fixed.md @@ -0,0 +1,24 @@ +--- +category: fixed +affected: + - src/CodeIndex/Indexer/SpanCharacterSearch.cs + - src/CodeIndex/Indexer/Symbols/SymbolExtractor.RepositoryMetadata.cs + - src/CodeIndex/Indexer/References/Languages/RepositoryMetadataReferenceExtractor.cs + - tests/CodeIndex.Tests/PerformanceTests.cs + - DEVELOPER_GUIDE.md + - TESTING_GUIDE.md +--- + +## English + +- **Reduced repository-metadata and manifest validation overhead** — + Long metadata candidates now use allocation-free span character scans, while + application manifests track dependency ancestry by XML depth instead of + rescanning an ancestor stack for every identity. + +## 日本語 + +- **repository metadata と manifest の validation overhead を削減しました** — + 長い metadata candidate は allocation-free な span character scan を使い、 + application manifest は identity ごとの ancestor stack 再走査ではなく XML depth + で dependency ancestry を追跡します。 diff --git a/src/CodeIndex/Indexer/References/Languages/RepositoryMetadataReferenceExtractor.cs b/src/CodeIndex/Indexer/References/Languages/RepositoryMetadataReferenceExtractor.cs index de1ab8b355..f81f14ff93 100644 --- a/src/CodeIndex/Indexer/References/Languages/RepositoryMetadataReferenceExtractor.cs +++ b/src/CodeIndex/Indexer/References/Languages/RepositoryMetadataReferenceExtractor.cs @@ -141,7 +141,7 @@ internal static List ExtractApplicationManifest( { var references = ReferenceExtractor.CreateReferenceList(maxReferenceCount, Math.Min(lines.Length, 64)); var seen = new ReferenceDedupeSet(); - var ancestors = new Stack(); + var dependencyAncestorDepth = 0; var elementCount = 0; SymbolRecord? manifestContainer = null; @@ -154,8 +154,8 @@ internal static List ExtractApplicationManifest( { if (reader.NodeType == XmlNodeType.EndElement) { - if (ancestors.Count > 0) - ancestors.Pop(); + if (IsManifestDependencyElement(reader.LocalName)) + dependencyAncestorDepth--; continue; } @@ -176,9 +176,7 @@ internal static List ExtractApplicationManifest( var elementName = reader.LocalName; var isDependencyIdentity = elementName.Equals("assemblyIdentity", StringComparison.OrdinalIgnoreCase) - && ancestors.Any(ancestor => - ancestor.Equals("dependentAssembly", StringComparison.OrdinalIgnoreCase) - || ancestor.Equals("dependency", StringComparison.OrdinalIgnoreCase)); + && dependencyAncestorDepth > 0; if (isDependencyIdentity) { AddManifestDependency( @@ -248,8 +246,11 @@ internal static List ExtractApplicationManifest( } } - if (!reader.IsEmptyElement) - ancestors.Push(elementName); + if (!reader.IsEmptyElement + && IsManifestDependencyElement(elementName)) + { + dependencyAncestorDepth++; + } } } catch (XmlException) @@ -260,6 +261,10 @@ internal static List ExtractApplicationManifest( return references; } + private static bool IsManifestDependencyElement(string elementName) + => elementName.Equals("dependentAssembly", StringComparison.OrdinalIgnoreCase) + || elementName.Equals("dependency", StringComparison.OrdinalIgnoreCase); + private static void AddAssignmentPathReferences( List references, ReferenceDedupeSet seen, @@ -506,8 +511,8 @@ private static bool TryNormalizeRepositoryPath( if (value.Length == 0 || value.Length > SymbolExtractor.StructuredDataMaxPathLength || value.Contains("://", StringComparison.Ordinal) - || !allowWhitespace && value.Any(char.IsWhiteSpace) - || value.Any(char.IsControl)) + || !allowWhitespace && SpanCharacterSearch.ContainsWhitespace(value) + || SpanCharacterSearch.ContainsControl(value)) { return false; } @@ -543,7 +548,7 @@ private static bool TryNormalizeRepositoryPath( if (!allowBarePattern && !value.Contains('/', StringComparison.Ordinal) - && !value.Any(character => character is '*' or '?' or '[' or '{') + && value.AsSpan().IndexOfAny("*?[{") < 0 && !HasFileLikeExtension(value)) { return false; @@ -577,7 +582,7 @@ private static void AddManifestDependency( || name.Contains("://", StringComparison.Ordinal) || name.Contains('/') || name.Contains('\\') - || name.Any(char.IsControl)) + || SpanCharacterSearch.ContainsControl(name)) return; var sourceIndex = context.IndexOf(name, StringComparison.Ordinal); diff --git a/src/CodeIndex/Indexer/SpanCharacterSearch.cs b/src/CodeIndex/Indexer/SpanCharacterSearch.cs new file mode 100644 index 0000000000..8dff2b43fd --- /dev/null +++ b/src/CodeIndex/Indexer/SpanCharacterSearch.cs @@ -0,0 +1,26 @@ +namespace CodeIndex.Indexer; + +internal static class SpanCharacterSearch +{ + internal static bool ContainsControl(ReadOnlySpan value) + { + for (var index = 0; index < value.Length; index++) + { + if (char.IsControl(value[index])) + return true; + } + + return false; + } + + internal static bool ContainsWhitespace(ReadOnlySpan value) + { + for (var index = 0; index < value.Length; index++) + { + if (char.IsWhiteSpace(value[index])) + return true; + } + + return false; + } +} diff --git a/src/CodeIndex/Indexer/Symbols/SymbolExtractor.RepositoryMetadata.cs b/src/CodeIndex/Indexer/Symbols/SymbolExtractor.RepositoryMetadata.cs index 42b5d85634..5243878335 100644 --- a/src/CodeIndex/Indexer/Symbols/SymbolExtractor.RepositoryMetadata.cs +++ b/src/CodeIndex/Indexer/Symbols/SymbolExtractor.RepositoryMetadata.cs @@ -331,12 +331,14 @@ internal static bool TryGetGitAttributesTokens( } if (patternEnd == 0 - || builder.Length == 0 - || builder.ToString().Any(char.IsControl)) + || builder.Length == 0) { return false; } - pattern = builder.ToString(); + var decodedPattern = builder.ToString(); + if (SpanCharacterSearch.ContainsControl(decodedPattern)) + return false; + pattern = decodedPattern; } else { diff --git a/tests/CodeIndex.Tests/PerformanceTests.cs b/tests/CodeIndex.Tests/PerformanceTests.cs index 1bb14d0c53..3375d7216a 100644 --- a/tests/CodeIndex.Tests/PerformanceTests.cs +++ b/tests/CodeIndex.Tests/PerformanceTests.cs @@ -559,6 +559,36 @@ public void HardwareScopeMembership_RepeatedIdentifierFiltering_DoesNotAllocate( $"Hardware scope membership allocated {allocatedBytes:N0} bytes"); } +#if NET8_0 + [Fact] +#else + [Fact(Skip = PracticalBudgetTestTarget.SecondaryTargetSkipReason)] +#endif + public void SpanCharacterSearch_RepeatedLongMetadataCandidates_DoesNotAllocate() + { + var candidate = new string('x', 4_096); + var matches = 0; + _ = SpanCharacterSearch.ContainsControl(candidate); + _ = SpanCharacterSearch.ContainsWhitespace(candidate); + + var allocatedBytes = MeasureAllocatedBytes(() => + { + for (var index = 0; index < 2_048; index++) + { + if (SpanCharacterSearch.ContainsControl(candidate) + || SpanCharacterSearch.ContainsWhitespace(candidate)) + { + matches++; + } + } + }); + + Assert.Equal(0, matches); + Assert.True( + allocatedBytes < 1_024, + $"Span character classification allocated {allocatedBytes:N0} bytes"); + } + #if NET8_0 [Fact] #else From 6e0300493df7db0e71cc102d0aac689d613e6feb Mon Sep 17 00:00:00 2001 From: Widthdom Date: Sun, 26 Jul 2026 17:23:39 +0900 Subject: [PATCH 07/30] Split source lines at exact capacity --- DEVELOPER_GUIDE.md | 7 ++++ TESTING_GUIDE.md | 2 ++ .../+source-line-splitting.fixed.md | 22 ++++++++++++ .../ReferenceExtractor.Configuration.cs | 2 +- src/CodeIndex/Indexer/SourceLineSplitter.cs | 33 ++++++++++++++++++ .../Indexer/Symbols/SymbolExtractor.cs | 2 +- tests/CodeIndex.Tests/PerformanceTests.cs | 34 +++++++++++++++++++ 7 files changed, 100 insertions(+), 2 deletions(-) create mode 100644 changelog.d/unreleased/+source-line-splitting.fixed.md create mode 100644 src/CodeIndex/Indexer/SourceLineSplitter.cs diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index 5546ad5a62..41951bfa2d 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -523,6 +523,10 @@ Repository metadata character validation uses `SpanCharacterSearch` instead of predicate-based enumeration, and application-manifest dependency ownership is tracked as XML depth rather than rescanning an ancestor stack for every `assemblyIdentity`. +All line-based symbol and reference extractors share `SourceLineSplitter`. +It counts newline boundaries once, allocates the exact result array, and then +materializes only the line strings that downstream scanners require; do not +restore separator-index arrays through `string.Split`. When structural masking turns a source line into whitespace, do not materialize a trimmed copy merely to discover that the line has no references. Preserve @@ -3531,6 +3535,9 @@ CUDA / GLSL / HLSL / Metal / WGSL の binding / resource scope は direct indexe repository metadata の character validation は predicate-based enumeration ではなく `SpanCharacterSearch` を使い、application manifest の dependency ownership は `assemblyIdentity` ごとの ancestor stack 再走査ではなく XML depth で追跡する。 +line-based symbol / reference extractor はすべて `SourceLineSplitter` を共有する。 +newline boundary を一度数えて exact result array を確保し、downstream scanner が必要とする +line string だけを実体化する。`string.Split` による separator-index array を戻してはならない。 構造マスクによって source line が空白だけになった場合、reference がないことを確認するため だけに trim 済み copy を実体化してはならない。documentation handling と、original line を diff --git a/TESTING_GUIDE.md b/TESTING_GUIDE.md index b75d8b80dc..2abfefb03a 100644 --- a/TESTING_GUIDE.md +++ b/TESTING_GUIDE.md @@ -582,6 +582,7 @@ Use `docs/test-doc-maintenance-plan.md` before moving oversized suites or adding `FunctionalSpanMembership_RepeatedCallFiltering_DoesNotAllocate` prevents per-match capturing-predicate allocations from returning to Erlang, OCAML, and Raku exclusion-range checks. `HardwareScopeMembership_RepeatedIdentifierFiltering_DoesNotAllocate` prevents per-identifier predicate closures from returning to shader binding/resource scope checks shared by CUDA, GLSL, HLSL, Metal, and WGSL; the corresponding HDL extraction regression covers shadowing semantics for Verilog, SystemVerilog, and VHDL. `SpanCharacterSearch_RepeatedLongMetadataCandidates_DoesNotAllocate` keeps control/whitespace validation allocation-free for long repository-metadata candidates; application-manifest regressions separately preserve dependency ancestry and local probing-path behavior. + `SourceLineSplitting_LargeFiles_AvoidsSeparatorIndexArrays` covers the shared all-language symbol/reference line splitter with 8,192 lines and fixes its exact output plus bounded allocation contract. `ReferenceExtraction_CSharpNoAliasDenseReferences_StaysWithinAllocationBudget` keeps 12,000 no-alias calls from paying for alias dedupe, `CSharpAliasCompaction_DenseDuplicates_StaysWithinAllocationBudget` guards pre-sized stable compaction after alias rewrites, and `MutualRecursion_DenseRepeatedQualifiedNames_StaysWithinAllocationBudget` covers repeated qualified C#- and Python-style cycle names without per-edge normalization strings. `ReusableStatSnapshot_OnePassMaterialization_StaysWithinAllocationBudget` warms a 1,024-row typed snapshot with its exact capacity, then prevents a second candidate collection or redundant path values from returning to the default `net8.0` path. `ReferenceBatchTransactions_RepositoryScaleAtomicFileScopeEliminatesControlledSqlScopes` models 321,352 references distributed over 856 files in the repository snapshot's five/six-batch shape and fixes the control-SQL contract at 5,009 public batch transaction scopes versus zero explicit atomic-file batch scopes; keep it deterministic and allocation-light instead of inserting all modeled rows or asserting elapsed time. For reference-line window performance audits, measure identical prebuilt reference rows against one-batch and 32-batch limits in alternating order, report both elapsed time and `GC.GetAllocatedBytesForCurrentThread`, and remove the timing harness after recording the result; end-to-end `--memory-trace` rebuilds remain corroborating evidence because extraction, graph finalization, and OS page-cache variance can dominate the persistence delta. @@ -1437,6 +1438,7 @@ dotnet test --filter "FullyQualifiedName~GitHelperTests" `FunctionalSpanMembership_RepeatedCallFiltering_DoesNotAllocate` は、Erlang、OCAML、Raku の exclusion-range 判定へ match ごとの capturing-predicate allocation が戻らないことを固定します。 `HardwareScopeMembership_RepeatedIdentifierFiltering_DoesNotAllocate` は、CUDA、GLSL、HLSL、Metal、WGSL が共有する shader binding / resource scope 判定へ identifier ごとの predicate closure が戻らないことを固定し、対応する HDL extraction regression が Verilog、SystemVerilog、VHDL の shadowing semantics をカバーします。 `SpanCharacterSearch_RepeatedLongMetadataCandidates_DoesNotAllocate` は、長い repository-metadata candidate の control / whitespace validation を allocation-free に固定し、application-manifest regression が dependency ancestry と local probing-path の意味論を別途維持します。 + `SourceLineSplitting_LargeFiles_AvoidsSeparatorIndexArrays` は、全言語の symbol / reference extraction が共有する line splitter を8,192行で検証し、exact output と bounded allocation contract を固定します。 `ReferenceExtraction_CSharpNoAliasDenseReferences_StaysWithinAllocationBudget` は alias のない 12,000 call が alias dedupe のコストを負わないこと、`CSharpAliasCompaction_DenseDuplicates_StaysWithinAllocationBudget` は alias rewrite 後の事前 capacity 付き stable compaction、`MutualRecursion_DenseRepeatedQualifiedNames_StaysWithinAllocationBudget` は C# / Python 形式の qualified cycle name が edge ごとの正規化文字列を作らないことを保証します。 `ReusableStatSnapshot_OnePassMaterialization_StaysWithinAllocationBudget` は exact capacity を渡した1,024行の typed snapshot を warm-up し、2つ目の候補 collection や重複 path value が通常の `net8.0` 経路へ戻らないようにします。 `ReferenceBatchTransactions_RepositoryScaleAtomicFileScopeEliminatesControlledSqlScopes` は321,352 refsを自己snapshotの5/6 batch形状で856 filesへ分配し、制御SQL契約をpublic batch transaction scope 5,009回対explicit atomic-file scope 0回に固定します。全model rowを挿入したり経過時間をassertしたりせず、deterministicでallocation-lightなまま維持してください。reference-line windowの性能監査では、同一の事前構築済みreference rowを1-batch上限と32-batch上限で交互に測り、経過時間と`GC.GetAllocatedBytesForCurrentThread`の両方を報告して、結果記録後にtiming harnessを削除します。end-to-endの`--memory-trace` rebuildは、extraction・graph finalize・OS page cacheの変動が永続化差を支配し得るため、補助証拠として扱ってください。 diff --git a/changelog.d/unreleased/+source-line-splitting.fixed.md b/changelog.d/unreleased/+source-line-splitting.fixed.md new file mode 100644 index 0000000000..f7d8ac8c98 --- /dev/null +++ b/changelog.d/unreleased/+source-line-splitting.fixed.md @@ -0,0 +1,22 @@ +--- +category: fixed +affected: + - src/CodeIndex/Indexer/SourceLineSplitter.cs + - src/CodeIndex/Indexer/Symbols/SymbolExtractor.cs + - src/CodeIndex/Indexer/References/ReferenceExtractor.Configuration.cs + - tests/CodeIndex.Tests/PerformanceTests.cs + - DEVELOPER_GUIDE.md + - TESTING_GUIDE.md +--- + +## English + +- **Reduced all-language extraction allocations for large source files** — + Symbol and reference extraction now share an exact-capacity line splitter + that avoids the temporary separator-index arrays created by generic splitting. + +## 日本語 + +- **巨大 source file の全言語 extraction allocation を削減しました** — + symbol / reference extraction は exact-capacity line splitter を共有し、generic split + が作る一時 separator-index array を回避します。 diff --git a/src/CodeIndex/Indexer/References/ReferenceExtractor.Configuration.cs b/src/CodeIndex/Indexer/References/ReferenceExtractor.Configuration.cs index e24a6c6c29..4a1faa21ed 100644 --- a/src/CodeIndex/Indexer/References/ReferenceExtractor.Configuration.cs +++ b/src/CodeIndex/Indexer/References/ReferenceExtractor.Configuration.cs @@ -48,7 +48,7 @@ public static partial class ReferenceExtractor ]; private static string[] SplitContentLines(string content) => - content.IndexOf('\n', StringComparison.Ordinal) < 0 ? [content] : content.Split('\n'); + SourceLineSplitter.Split(content); internal static ReferenceExtractionSafetyLimits? SafetyLimitsForTesting { diff --git a/src/CodeIndex/Indexer/SourceLineSplitter.cs b/src/CodeIndex/Indexer/SourceLineSplitter.cs new file mode 100644 index 0000000000..ddac0f4269 --- /dev/null +++ b/src/CodeIndex/Indexer/SourceLineSplitter.cs @@ -0,0 +1,33 @@ +namespace CodeIndex.Indexer; + +internal static class SourceLineSplitter +{ + internal static string[] Split(string content) + { + var firstLineBreak = content.IndexOf('\n', StringComparison.Ordinal); + if (firstLineBreak < 0) + return [content]; + + var lineCount = 2; + for (var index = firstLineBreak + 1; index < content.Length; index++) + { + if (content[index] == '\n') + lineCount++; + } + + var lines = new string[lineCount]; + var lineIndex = 0; + var lineStart = 0; + for (var index = firstLineBreak; index < content.Length; index++) + { + if (content[index] != '\n') + continue; + + lines[lineIndex++] = content[lineStart..index]; + lineStart = index + 1; + } + + lines[lineIndex] = content[lineStart..]; + return lines; + } +} diff --git a/src/CodeIndex/Indexer/Symbols/SymbolExtractor.cs b/src/CodeIndex/Indexer/Symbols/SymbolExtractor.cs index 0d861febc2..b31e4e8abf 100644 --- a/src/CodeIndex/Indexer/Symbols/SymbolExtractor.cs +++ b/src/CodeIndex/Indexer/Symbols/SymbolExtractor.cs @@ -17,7 +17,7 @@ public static partial class SymbolExtractor private const int SymbolListInitialCapacityMax = 1024; private static string[] SplitContentLines(string content) => - content.IndexOf('\n', StringComparison.Ordinal) < 0 ? [content] : content.Split('\n'); + SourceLineSplitter.Split(content); private static List CreateSymbolListForLines(int lineCount) { diff --git a/tests/CodeIndex.Tests/PerformanceTests.cs b/tests/CodeIndex.Tests/PerformanceTests.cs index 3375d7216a..73a96ee5a3 100644 --- a/tests/CodeIndex.Tests/PerformanceTests.cs +++ b/tests/CodeIndex.Tests/PerformanceTests.cs @@ -589,6 +589,40 @@ public void SpanCharacterSearch_RepeatedLongMetadataCandidates_DoesNotAllocate() $"Span character classification allocated {allocatedBytes:N0} bytes"); } +#if NET8_0 + [Fact] +#else + [Fact(Skip = PracticalBudgetTestTarget.SecondaryTargetSkipReason)] +#endif + public void SourceLineSplitting_LargeFiles_AvoidsSeparatorIndexArrays() + { + Assert.Equal([""], SourceLineSplitter.Split(string.Empty)); + Assert.Equal(["line", ""], SourceLineSplitter.Split("line\n")); + Assert.Equal(["first", "", "third"], SourceLineSplitter.Split("first\n\nthird")); + + var content = string.Join( + '\n', + Enumerable.Range(0, 8_192) + .Select(index => $"line_{index:D5}_payload")); + _ = SourceLineSplitter.Split(content); + + string[]? lines = null; + var allocatedBytes = MeasureAllocatedBytes( + () => lines = SourceLineSplitter.Split(content)); + var genericSplitAllocatedBytes = MeasureAllocatedBytes( + () => lines = content.Split('\n')); + + Assert.Equal(8_192, lines!.Length); + Assert.Equal("line_00000_payload", lines[0]); + Assert.Equal("line_08191_payload", lines[^1]); + Assert.True( + allocatedBytes < 610_000, + $"Source line splitting allocated {allocatedBytes:N0} bytes"); + Assert.True( + allocatedBytes + 50_000 < genericSplitAllocatedBytes, + $"Source line splitting allocated {allocatedBytes:N0} bytes versus {genericSplitAllocatedBytes:N0} bytes for generic splitting"); + } + #if NET8_0 [Fact] #else From 49e90df507ab5806074b5e1c38e0f4c54319c451 Mon Sep 17 00:00:00 2001 From: Widthdom Date: Sun, 26 Jul 2026 17:25:46 +0900 Subject: [PATCH 08/30] Reuse prepared-line whitespace classification --- DEVELOPER_GUIDE.md | 5 +++++ .../+prepared-line-whitespace.fixed.md | 19 +++++++++++++++++++ .../ReferenceExtractor.CoreReferenceLoop.cs | 8 ++++++-- 3 files changed, 30 insertions(+), 2 deletions(-) create mode 100644 changelog.d/unreleased/+prepared-line-whitespace.fixed.md diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index 41951bfa2d..c802df6867 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -533,6 +533,9 @@ a trimmed copy merely to discover that the line has no references. Preserve documentation handling and the build-automation and markup paths that inspect the original line, but let ordinary C#, Java, and JavaScript / TypeScript code paths skip masked multiline payloads before creating a reference context. +Classify prepared-line whitespace once per core-loop iteration and reuse that +result for both special-line dispatch and the ordinary empty-line path; masked +payload lines can be thousands of characters long. C / C++ header disambiguation operates on bounded lexical samples. Walk those samples with spans and newline indices; splitting a sample into a line array @@ -3543,6 +3546,8 @@ line string だけを実体化する。`string.Split` による separator-index だけに trim 済み copy を実体化してはならない。documentation handling と、original line を 検査する build-automation / markup 経路は維持しつつ、通常の C#、Java、 JavaScript / TypeScript 経路では reference context を作る前に multiline payload を skip する。 +prepared-line の whitespace 判定は core-loop iteration ごとに一度だけ行い、special-line +dispatch と通常の empty-line 経路で共有する。masked payload line は数千文字になり得る。 C / C++ header の曖昧性解決は bounded lexical sample 上で行う。sample は span と newline index で走査し、line array に split してはならない。split は sample 内の全行を一時的に複製し、 diff --git a/changelog.d/unreleased/+prepared-line-whitespace.fixed.md b/changelog.d/unreleased/+prepared-line-whitespace.fixed.md new file mode 100644 index 0000000000..89c862c1d9 --- /dev/null +++ b/changelog.d/unreleased/+prepared-line-whitespace.fixed.md @@ -0,0 +1,19 @@ +--- +category: fixed +affected: + - src/CodeIndex/Indexer/References/ReferenceExtractor.CoreReferenceLoop.cs + - DEVELOPER_GUIDE.md +--- + +## English + +- **Avoided duplicate full-line whitespace scans during reference extraction** — + Every supported language now classifies each prepared line once and reuses + the result across special-line and ordinary empty-line dispatch, which is + especially important for long structurally masked payloads. + +## 日本語 + +- **reference extraction の重複した全行 whitespace scan を解消しました** — + 全対応言語で prepared line を一度だけ判定し、special-line と通常の empty-line dispatch + で結果を共有するため、長い構造マスク済み payload の再走査を避けます。 diff --git a/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreReferenceLoop.cs b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreReferenceLoop.cs index af69bcecf8..d154efecd1 100644 --- a/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreReferenceLoop.cs +++ b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreReferenceLoop.cs @@ -143,6 +143,8 @@ private static CSharpMultiLineTypePatternState EmitCoreReferenceLines( i, originalLine); var preparedLine = languageLines.PreparedLine; + var preparedLineIsWhiteSpace = + string.IsNullOrWhiteSpace(preparedLine); var originalLineForLanguage = languageLines.OriginalLineForLanguage; var csharpAttrRangesOnLine = csharpAttrRanges?[i]; @@ -153,13 +155,14 @@ private static CSharpMultiLineTypePatternState EmitCoreReferenceLines( i, originalLine, preparedLine, + preparedLineIsWhiteSpace, csharpAttrRangesOnLine, out var sourceContext)) { continue; } - if (string.IsNullOrWhiteSpace(preparedLine)) + if (preparedLineIsWhiteSpace) { if (language == "csharp" && (pendingCSharpMultiLineTypePattern.WaitingForHead @@ -527,6 +530,7 @@ private static bool EmitCoreDocumentationAndSpecialLineReferences( int lineIndex, string originalLine, string preparedLine, + bool preparedLineIsWhiteSpace, List<(int start, int end)>? csharpAttributeRangesOnLine, out string sourceContext) { @@ -585,7 +589,7 @@ private static bool EmitCoreDocumentationAndSpecialLineReferences( ref state.PhpDocblockPropertyNames); } - if (string.IsNullOrWhiteSpace(preparedLine) + if (preparedLineIsWhiteSpace && request.Language is not ("cmake" or "justfile" From 9ddf12dfa1a9f8554ef3f9f93f9f948b55895245 Mon Sep 17 00:00:00 2001 From: Widthdom Date: Sun, 26 Jul 2026 17:28:50 +0900 Subject: [PATCH 09/30] Compare functional terminators on spans --- DEVELOPER_GUIDE.md | 5 +++ TESTING_GUIDE.md | 2 + .../+functional-terminator-spans.fixed.md | 21 ++++++++++ ...tionalLanguageReferenceExtractor.Erlang.cs | 2 +- .../FunctionalLanguageReferenceExtractor.cs | 22 +++++++++-- tests/CodeIndex.Tests/PerformanceTests.cs | 38 +++++++++++++++++++ 6 files changed, 86 insertions(+), 4 deletions(-) create mode 100644 changelog.d/unreleased/+functional-terminator-spans.fixed.md diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index c802df6867..e7b67d3213 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -523,6 +523,9 @@ Repository metadata character validation uses `SpanCharacterSearch` instead of predicate-based enumeration, and application-manifest dependency ownership is tracked as XML depth rather than rescanning an ancestor stack for every `assemblyIdentity`. +State-machine sentinel checks must also stay on spans. Erlang specification and +callable terminators plus Raku heredoc terminators trim views of the original +line without materializing padded copies. All line-based symbol and reference extractors share `SourceLineSplitter`. It counts newline boundaries once, allocates the exact result array, and then materializes only the line strings that downstream scanners require; do not @@ -3538,6 +3541,8 @@ CUDA / GLSL / HLSL / Metal / WGSL の binding / resource scope は direct indexe repository metadata の character validation は predicate-based enumeration ではなく `SpanCharacterSearch` を使い、application manifest の dependency ownership は `assemblyIdentity` ごとの ancestor stack 再走査ではなく XML depth で追跡する。 +state-machine の sentinel 判定も span 上で行う。Erlang specification / callable terminator +と Raku heredoc terminator は、padding を含む copy を実体化せず original line の view を trim する。 line-based symbol / reference extractor はすべて `SourceLineSplitter` を共有する。 newline boundary を一度数えて exact result array を確保し、downstream scanner が必要とする line string だけを実体化する。`string.Split` による separator-index array を戻してはならない。 diff --git a/TESTING_GUIDE.md b/TESTING_GUIDE.md index 2abfefb03a..765f9ab963 100644 --- a/TESTING_GUIDE.md +++ b/TESTING_GUIDE.md @@ -583,6 +583,7 @@ Use `docs/test-doc-maintenance-plan.md` before moving oversized suites or adding `HardwareScopeMembership_RepeatedIdentifierFiltering_DoesNotAllocate` prevents per-identifier predicate closures from returning to shader binding/resource scope checks shared by CUDA, GLSL, HLSL, Metal, and WGSL; the corresponding HDL extraction regression covers shadowing semantics for Verilog, SystemVerilog, and VHDL. `SpanCharacterSearch_RepeatedLongMetadataCandidates_DoesNotAllocate` keeps control/whitespace validation allocation-free for long repository-metadata candidates; application-manifest regressions separately preserve dependency ancestry and local probing-path behavior. `SourceLineSplitting_LargeFiles_AvoidsSeparatorIndexArrays` covers the shared all-language symbol/reference line splitter with 8,192 lines and fixes its exact output plus bounded allocation contract. + `FunctionalTerminatorChecks_LongPaddedLines_DoNotAllocate` keeps Erlang specification/callable and Raku heredoc state-machine sentinels on trimmed spans rather than padded string copies. `ReferenceExtraction_CSharpNoAliasDenseReferences_StaysWithinAllocationBudget` keeps 12,000 no-alias calls from paying for alias dedupe, `CSharpAliasCompaction_DenseDuplicates_StaysWithinAllocationBudget` guards pre-sized stable compaction after alias rewrites, and `MutualRecursion_DenseRepeatedQualifiedNames_StaysWithinAllocationBudget` covers repeated qualified C#- and Python-style cycle names without per-edge normalization strings. `ReusableStatSnapshot_OnePassMaterialization_StaysWithinAllocationBudget` warms a 1,024-row typed snapshot with its exact capacity, then prevents a second candidate collection or redundant path values from returning to the default `net8.0` path. `ReferenceBatchTransactions_RepositoryScaleAtomicFileScopeEliminatesControlledSqlScopes` models 321,352 references distributed over 856 files in the repository snapshot's five/six-batch shape and fixes the control-SQL contract at 5,009 public batch transaction scopes versus zero explicit atomic-file batch scopes; keep it deterministic and allocation-light instead of inserting all modeled rows or asserting elapsed time. For reference-line window performance audits, measure identical prebuilt reference rows against one-batch and 32-batch limits in alternating order, report both elapsed time and `GC.GetAllocatedBytesForCurrentThread`, and remove the timing harness after recording the result; end-to-end `--memory-trace` rebuilds remain corroborating evidence because extraction, graph finalization, and OS page-cache variance can dominate the persistence delta. @@ -1439,6 +1440,7 @@ dotnet test --filter "FullyQualifiedName~GitHelperTests" `HardwareScopeMembership_RepeatedIdentifierFiltering_DoesNotAllocate` は、CUDA、GLSL、HLSL、Metal、WGSL が共有する shader binding / resource scope 判定へ identifier ごとの predicate closure が戻らないことを固定し、対応する HDL extraction regression が Verilog、SystemVerilog、VHDL の shadowing semantics をカバーします。 `SpanCharacterSearch_RepeatedLongMetadataCandidates_DoesNotAllocate` は、長い repository-metadata candidate の control / whitespace validation を allocation-free に固定し、application-manifest regression が dependency ancestry と local probing-path の意味論を別途維持します。 `SourceLineSplitting_LargeFiles_AvoidsSeparatorIndexArrays` は、全言語の symbol / reference extraction が共有する line splitter を8,192行で検証し、exact output と bounded allocation contract を固定します。 + `FunctionalTerminatorChecks_LongPaddedLines_DoNotAllocate` は、Erlang specification / callable と Raku heredoc の state-machine sentinel を、padding 付き string copy ではなく trimmed span 上に固定します。 `ReferenceExtraction_CSharpNoAliasDenseReferences_StaysWithinAllocationBudget` は alias のない 12,000 call が alias dedupe のコストを負わないこと、`CSharpAliasCompaction_DenseDuplicates_StaysWithinAllocationBudget` は alias rewrite 後の事前 capacity 付き stable compaction、`MutualRecursion_DenseRepeatedQualifiedNames_StaysWithinAllocationBudget` は C# / Python 形式の qualified cycle name が edge ごとの正規化文字列を作らないことを保証します。 `ReusableStatSnapshot_OnePassMaterialization_StaysWithinAllocationBudget` は exact capacity を渡した1,024行の typed snapshot を warm-up し、2つ目の候補 collection や重複 path value が通常の `net8.0` 経路へ戻らないようにします。 `ReferenceBatchTransactions_RepositoryScaleAtomicFileScopeEliminatesControlledSqlScopes` は321,352 refsを自己snapshotの5/6 batch形状で856 filesへ分配し、制御SQL契約をpublic batch transaction scope 5,009回対explicit atomic-file scope 0回に固定します。全model rowを挿入したり経過時間をassertしたりせず、deterministicでallocation-lightなまま維持してください。reference-line windowの性能監査では、同一の事前構築済みreference rowを1-batch上限と32-batch上限で交互に測り、経過時間と`GC.GetAllocatedBytesForCurrentThread`の両方を報告して、結果記録後にtiming harnessを削除します。end-to-endの`--memory-trace` rebuildは、extraction・graph finalize・OS page cacheの変動が永続化差を支配し得るため、補助証拠として扱ってください。 diff --git a/changelog.d/unreleased/+functional-terminator-spans.fixed.md b/changelog.d/unreleased/+functional-terminator-spans.fixed.md new file mode 100644 index 0000000000..69131e02c5 --- /dev/null +++ b/changelog.d/unreleased/+functional-terminator-spans.fixed.md @@ -0,0 +1,21 @@ +--- +category: fixed +affected: + - src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.cs + - src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Erlang.cs + - tests/CodeIndex.Tests/PerformanceTests.cs + - DEVELOPER_GUIDE.md + - TESTING_GUIDE.md +--- + +## English + +- **Removed padded-line copies from functional-language state machines** — + Erlang specification/callable terminators and Raku heredoc terminators now + compare trimmed spans, avoiding a new string for every padded sentinel line. + +## 日本語 + +- **functional-language state machine の padded-line copy を解消しました** — + Erlang specification / callable terminator と Raku heredoc terminator は trimmed span + を比較し、padding 付き sentinel line ごとの新しい string を回避します。 diff --git a/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Erlang.cs b/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Erlang.cs index b69d743972..85c552da71 100644 --- a/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Erlang.cs +++ b/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Erlang.cs @@ -23,7 +23,7 @@ private static void EmitErlangReferences( state.ErlangSpecificationMode = true; if (state.ErlangSpecificationMode) { - if (line.TrimEnd().EndsWith(".", StringComparison.Ordinal)) + if (TrimmedFunctionalLineEndsWith(line, '.')) state.ErlangSpecificationMode = false; return; } diff --git a/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.cs b/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.cs index f10b4ddbb1..561688e362 100644 --- a/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.cs +++ b/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.cs @@ -181,6 +181,19 @@ internal static bool OverlapsFunctionalSpan( return false; } + internal static bool TrimmedFunctionalLineEndsWith( + ReadOnlySpan line, + char suffix) + { + line = line.TrimEnd(); + return !line.IsEmpty && line[^1] == suffix; + } + + internal static bool TrimmedFunctionalLineEquals( + ReadOnlySpan line, + string expected) + => line.TrimEnd().Equals(expected, StringComparison.Ordinal); + private static List ExtractFunctionalLanguageReferences(ReferenceExtractionContext request) { if (!TryPrepareReferenceLines( @@ -389,7 +402,8 @@ private static void AdvanceFunctionalCallableState( } break; case "erlang": - if (state.ActiveCallable != null && maskedLine.TrimEnd().EndsWith(".", StringComparison.Ordinal)) + if (state.ActiveCallable != null + && TrimmedFunctionalLineEndsWith(maskedLine, '.')) state.ActiveCallable = null; break; case "raku": @@ -431,10 +445,12 @@ private static string MaskFunctionalReferenceLine( { if (language == "raku") { - var trimmed = line.TrimStart(); + var trimmed = line.AsSpan().TrimStart(); if (state.RakuHeredocTerminator != null) { - if (string.Equals(trimmed.TrimEnd(), state.RakuHeredocTerminator, StringComparison.Ordinal)) + if (TrimmedFunctionalLineEquals( + trimmed, + state.RakuHeredocTerminator)) state.RakuHeredocTerminator = null; return new string(' ', line.Length); } diff --git a/tests/CodeIndex.Tests/PerformanceTests.cs b/tests/CodeIndex.Tests/PerformanceTests.cs index 73a96ee5a3..6c4f939583 100644 --- a/tests/CodeIndex.Tests/PerformanceTests.cs +++ b/tests/CodeIndex.Tests/PerformanceTests.cs @@ -623,6 +623,44 @@ public void SourceLineSplitting_LargeFiles_AvoidsSeparatorIndexArrays() $"Source line splitting allocated {allocatedBytes:N0} bytes versus {genericSplitAllocatedBytes:N0} bytes for generic splitting"); } +#if NET8_0 + [Fact] +#else + [Fact(Skip = PracticalBudgetTestTarget.SecondaryTargetSkipReason)] +#endif + public void FunctionalTerminatorChecks_LongPaddedLines_DoNotAllocate() + { + var periodLine = $"value.{new string(' ', 4_096)}"; + var heredocLine = $"END{new string(' ', 4_096)}"; + var matchCount = 0; + _ = ReferenceExtractor.TrimmedFunctionalLineEndsWith(periodLine, '.'); + _ = ReferenceExtractor.TrimmedFunctionalLineEquals(heredocLine, "END"); + + var allocatedBytes = MeasureAllocatedBytes(() => + { + for (var index = 0; index < 4_096; index++) + { + if (ReferenceExtractor.TrimmedFunctionalLineEndsWith( + periodLine, + '.')) + { + matchCount++; + } + if (ReferenceExtractor.TrimmedFunctionalLineEquals( + heredocLine, + "END")) + { + matchCount++; + } + } + }); + + Assert.Equal(8_192, matchCount); + Assert.True( + allocatedBytes < 1_024, + $"Functional terminator checks allocated {allocatedBytes:N0} bytes"); + } + #if NET8_0 [Fact] #else From 299854e78e5ead7afcf030daf387df4a0db47359 Mon Sep 17 00:00:00 2001 From: Widthdom Date: Sun, 26 Jul 2026 17:31:49 +0900 Subject: [PATCH 10/30] Share trimmed suffix checks across scanners --- DEVELOPER_GUIDE.md | 5 +++ TESTING_GUIDE.md | 2 ++ .../unreleased/+trimmed-suffix-spans.fixed.md | 24 ++++++++++++++ ...ferenceExtractor.AnimationsAndSelectors.cs | 3 +- .../FunctionalLanguageReferenceExtractor.cs | 7 ++-- src/CodeIndex/Indexer/SpanCharacterSearch.cs | 13 ++++++++ .../Symbols/SymbolExtractor.CSharpScanner.cs | 4 ++- .../Indexer/Symbols/SymbolExtractor.Java.cs | 2 +- tests/CodeIndex.Tests/PerformanceTests.cs | 33 +++++++++++++++++++ 9 files changed, 85 insertions(+), 8 deletions(-) create mode 100644 changelog.d/unreleased/+trimmed-suffix-spans.fixed.md diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index e7b67d3213..223291ad61 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -526,6 +526,9 @@ tracked as XML depth rather than rescanning an ancestor stack for every State-machine sentinel checks must also stay on spans. Erlang specification and callable terminators plus Raku heredoc terminators trim views of the original line without materializing padded copies. +`SpanCharacterSearch.EndsWithAfterTrim` is the shared suffix primitive for +these sentinels and for CSS selector continuations plus C# / Java body-less +declaration termination. All line-based symbol and reference extractors share `SourceLineSplitter`. It counts newline boundaries once, allocates the exact result array, and then materializes only the line strings that downstream scanners require; do not @@ -3543,6 +3546,8 @@ repository metadata の character validation は predicate-based enumeration で `assemblyIdentity` ごとの ancestor stack 再走査ではなく XML depth で追跡する。 state-machine の sentinel 判定も span 上で行う。Erlang specification / callable terminator と Raku heredoc terminator は、padding を含む copy を実体化せず original line の view を trim する。 +`SpanCharacterSearch.EndsWithAfterTrim` はこれらの sentinel に加え、CSS selector continuation +と C# / Java の body-less declaration termination が共有する suffix primitive である。 line-based symbol / reference extractor はすべて `SourceLineSplitter` を共有する。 newline boundary を一度数えて exact result array を確保し、downstream scanner が必要とする line string だけを実体化する。`string.Split` による separator-index array を戻してはならない。 diff --git a/TESTING_GUIDE.md b/TESTING_GUIDE.md index 765f9ab963..1003290ca3 100644 --- a/TESTING_GUIDE.md +++ b/TESTING_GUIDE.md @@ -584,6 +584,7 @@ Use `docs/test-doc-maintenance-plan.md` before moving oversized suites or adding `SpanCharacterSearch_RepeatedLongMetadataCandidates_DoesNotAllocate` keeps control/whitespace validation allocation-free for long repository-metadata candidates; application-manifest regressions separately preserve dependency ancestry and local probing-path behavior. `SourceLineSplitting_LargeFiles_AvoidsSeparatorIndexArrays` covers the shared all-language symbol/reference line splitter with 8,192 lines and fixes its exact output plus bounded allocation contract. `FunctionalTerminatorChecks_LongPaddedLines_DoNotAllocate` keeps Erlang specification/callable and Raku heredoc state-machine sentinels on trimmed spans rather than padded string copies. + `TrimmedSuffixChecks_LongDeclarationLines_DoNotAllocate` keeps CSS selector continuations and C# / Java body-less declaration suffix checks allocation-free on long padded lines. `ReferenceExtraction_CSharpNoAliasDenseReferences_StaysWithinAllocationBudget` keeps 12,000 no-alias calls from paying for alias dedupe, `CSharpAliasCompaction_DenseDuplicates_StaysWithinAllocationBudget` guards pre-sized stable compaction after alias rewrites, and `MutualRecursion_DenseRepeatedQualifiedNames_StaysWithinAllocationBudget` covers repeated qualified C#- and Python-style cycle names without per-edge normalization strings. `ReusableStatSnapshot_OnePassMaterialization_StaysWithinAllocationBudget` warms a 1,024-row typed snapshot with its exact capacity, then prevents a second candidate collection or redundant path values from returning to the default `net8.0` path. `ReferenceBatchTransactions_RepositoryScaleAtomicFileScopeEliminatesControlledSqlScopes` models 321,352 references distributed over 856 files in the repository snapshot's five/six-batch shape and fixes the control-SQL contract at 5,009 public batch transaction scopes versus zero explicit atomic-file batch scopes; keep it deterministic and allocation-light instead of inserting all modeled rows or asserting elapsed time. For reference-line window performance audits, measure identical prebuilt reference rows against one-batch and 32-batch limits in alternating order, report both elapsed time and `GC.GetAllocatedBytesForCurrentThread`, and remove the timing harness after recording the result; end-to-end `--memory-trace` rebuilds remain corroborating evidence because extraction, graph finalization, and OS page-cache variance can dominate the persistence delta. @@ -1441,6 +1442,7 @@ dotnet test --filter "FullyQualifiedName~GitHelperTests" `SpanCharacterSearch_RepeatedLongMetadataCandidates_DoesNotAllocate` は、長い repository-metadata candidate の control / whitespace validation を allocation-free に固定し、application-manifest regression が dependency ancestry と local probing-path の意味論を別途維持します。 `SourceLineSplitting_LargeFiles_AvoidsSeparatorIndexArrays` は、全言語の symbol / reference extraction が共有する line splitter を8,192行で検証し、exact output と bounded allocation contract を固定します。 `FunctionalTerminatorChecks_LongPaddedLines_DoNotAllocate` は、Erlang specification / callable と Raku heredoc の state-machine sentinel を、padding 付き string copy ではなく trimmed span 上に固定します。 + `TrimmedSuffixChecks_LongDeclarationLines_DoNotAllocate` は、長い padding 付き行に対する CSS selector continuation と C# / Java body-less declaration の suffix 判定を allocation-free に固定します。 `ReferenceExtraction_CSharpNoAliasDenseReferences_StaysWithinAllocationBudget` は alias のない 12,000 call が alias dedupe のコストを負わないこと、`CSharpAliasCompaction_DenseDuplicates_StaysWithinAllocationBudget` は alias rewrite 後の事前 capacity 付き stable compaction、`MutualRecursion_DenseRepeatedQualifiedNames_StaysWithinAllocationBudget` は C# / Python 形式の qualified cycle name が edge ごとの正規化文字列を作らないことを保証します。 `ReusableStatSnapshot_OnePassMaterialization_StaysWithinAllocationBudget` は exact capacity を渡した1,024行の typed snapshot を warm-up し、2つ目の候補 collection や重複 path value が通常の `net8.0` 経路へ戻らないようにします。 `ReferenceBatchTransactions_RepositoryScaleAtomicFileScopeEliminatesControlledSqlScopes` は321,352 refsを自己snapshotの5/6 batch形状で856 filesへ分配し、制御SQL契約をpublic batch transaction scope 5,009回対explicit atomic-file scope 0回に固定します。全model rowを挿入したり経過時間をassertしたりせず、deterministicでallocation-lightなまま維持してください。reference-line windowの性能監査では、同一の事前構築済みreference rowを1-batch上限と32-batch上限で交互に測り、経過時間と`GC.GetAllocatedBytesForCurrentThread`の両方を報告して、結果記録後にtiming harnessを削除します。end-to-endの`--memory-trace` rebuildは、extraction・graph finalize・OS page cacheの変動が永続化差を支配し得るため、補助証拠として扱ってください。 diff --git a/changelog.d/unreleased/+trimmed-suffix-spans.fixed.md b/changelog.d/unreleased/+trimmed-suffix-spans.fixed.md new file mode 100644 index 0000000000..f945aaedcd --- /dev/null +++ b/changelog.d/unreleased/+trimmed-suffix-spans.fixed.md @@ -0,0 +1,24 @@ +--- +category: fixed +affected: + - src/CodeIndex/Indexer/SpanCharacterSearch.cs + - src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.cs + - src/CodeIndex/Indexer/References/Languages/CssReferenceExtractor.AnimationsAndSelectors.cs + - src/CodeIndex/Indexer/Symbols/SymbolExtractor.CSharpScanner.cs + - src/CodeIndex/Indexer/Symbols/SymbolExtractor.Java.cs + - tests/CodeIndex.Tests/PerformanceTests.cs + - DEVELOPER_GUIDE.md + - TESTING_GUIDE.md +--- + +## English + +- **Removed padded suffix copies across declaration scanners** — CSS selector + continuations, C# and Java body-less declarations, and functional-language + sentinels now share allocation-free trimmed span comparisons. + +## 日本語 + +- **declaration scanner 横断で padded suffix copy を解消しました** — + CSS selector continuation、C# / Java body-less declaration、functional-language + sentinel は allocation-free な trimmed span 比較を共有します。 diff --git a/src/CodeIndex/Indexer/References/Languages/CssReferenceExtractor.AnimationsAndSelectors.cs b/src/CodeIndex/Indexer/References/Languages/CssReferenceExtractor.AnimationsAndSelectors.cs index 77e069c7b1..75fc87a543 100644 --- a/src/CodeIndex/Indexer/References/Languages/CssReferenceExtractor.AnimationsAndSelectors.cs +++ b/src/CodeIndex/Indexer/References/Languages/CssReferenceExtractor.AnimationsAndSelectors.cs @@ -217,7 +217,8 @@ private static void EmitCssClassSelectorReferences( // ID セレクタ (`#name`) は `#fff` 等の color literal とパターンが衝突するため、 // セレクタ位置のセグメントでのみ参照を発行する。セグメントが本行内で `{` で // 終わる場合、または行末カンマで selector list が継続する場合をセレクタ位置とみなす。 - var isSelectorContinuationLine = preparedLine.TrimEnd().EndsWith(','); + var isSelectorContinuationLine = + SpanCharacterSearch.EndsWithAfterTrim(preparedLine, ','); var segmentStart = 0; while (segmentStart < preparedLine.Length) { diff --git a/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.cs b/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.cs index 561688e362..d08e8faece 100644 --- a/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.cs +++ b/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.cs @@ -184,15 +184,12 @@ internal static bool OverlapsFunctionalSpan( internal static bool TrimmedFunctionalLineEndsWith( ReadOnlySpan line, char suffix) - { - line = line.TrimEnd(); - return !line.IsEmpty && line[^1] == suffix; - } + => SpanCharacterSearch.EndsWithAfterTrim(line, suffix); internal static bool TrimmedFunctionalLineEquals( ReadOnlySpan line, string expected) - => line.TrimEnd().Equals(expected, StringComparison.Ordinal); + => SpanCharacterSearch.EqualsAfterTrim(line, expected); private static List ExtractFunctionalLanguageReferences(ReferenceExtractionContext request) { diff --git a/src/CodeIndex/Indexer/SpanCharacterSearch.cs b/src/CodeIndex/Indexer/SpanCharacterSearch.cs index 8dff2b43fd..b9c744025b 100644 --- a/src/CodeIndex/Indexer/SpanCharacterSearch.cs +++ b/src/CodeIndex/Indexer/SpanCharacterSearch.cs @@ -23,4 +23,17 @@ internal static bool ContainsWhitespace(ReadOnlySpan value) return false; } + + internal static bool EndsWithAfterTrim( + ReadOnlySpan value, + char suffix) + { + value = value.TrimEnd(); + return !value.IsEmpty && value[^1] == suffix; + } + + internal static bool EqualsAfterTrim( + ReadOnlySpan value, + string expected) + => value.TrimEnd().Equals(expected, StringComparison.Ordinal); } diff --git a/src/CodeIndex/Indexer/Symbols/SymbolExtractor.CSharpScanner.cs b/src/CodeIndex/Indexer/Symbols/SymbolExtractor.CSharpScanner.cs index 35ca46d503..c11510a381 100644 --- a/src/CodeIndex/Indexer/Symbols/SymbolExtractor.CSharpScanner.cs +++ b/src/CodeIndex/Indexer/Symbols/SymbolExtractor.CSharpScanner.cs @@ -428,7 +428,9 @@ private static (int EndLine, int? BodyStartLine, int? BodyEndLine) FindCSharpBra } } - if (!opened && !expressionBody && scanLine.TrimEnd().EndsWith(';')) + if (!opened + && !expressionBody + && SpanCharacterSearch.EndsWithAfterTrim(scanLine, ';')) return (startIndex + 1, null, null); } diff --git a/src/CodeIndex/Indexer/Symbols/SymbolExtractor.Java.cs b/src/CodeIndex/Indexer/Symbols/SymbolExtractor.Java.cs index 11ee125c16..a8dd452c5f 100644 --- a/src/CodeIndex/Indexer/Symbols/SymbolExtractor.Java.cs +++ b/src/CodeIndex/Indexer/Symbols/SymbolExtractor.Java.cs @@ -1533,7 +1533,7 @@ internal static (int EndLine, int? BodyStartLine, int? BodyEndLine) FindJavaBrac if (!opened && mode == JavaScanMode.Normal && parenDepth == 0 && bracketDepth == 0 && angleDepth == 0 - && line.TrimEnd().EndsWith(';')) + && SpanCharacterSearch.EndsWithAfterTrim(line, ';')) return (startIndex + 1, null, null); } diff --git a/tests/CodeIndex.Tests/PerformanceTests.cs b/tests/CodeIndex.Tests/PerformanceTests.cs index 6c4f939583..971a28801c 100644 --- a/tests/CodeIndex.Tests/PerformanceTests.cs +++ b/tests/CodeIndex.Tests/PerformanceTests.cs @@ -661,6 +661,39 @@ public void FunctionalTerminatorChecks_LongPaddedLines_DoNotAllocate() $"Functional terminator checks allocated {allocatedBytes:N0} bytes"); } +#if NET8_0 + [Fact] +#else + [Fact(Skip = PracticalBudgetTestTarget.SecondaryTargetSkipReason)] +#endif + public void TrimmedSuffixChecks_LongDeclarationLines_DoNotAllocate() + { + var semicolonLine = $"declaration;{new string(' ', 4_096)}"; + var commaLine = $"selector,{new string(' ', 4_096)}"; + var matchCount = 0; + _ = SpanCharacterSearch.EndsWithAfterTrim(semicolonLine, ';'); + + var allocatedBytes = MeasureAllocatedBytes(() => + { + for (var index = 0; index < 4_096; index++) + { + if (SpanCharacterSearch.EndsWithAfterTrim( + semicolonLine, + ';')) + { + matchCount++; + } + if (SpanCharacterSearch.EndsWithAfterTrim(commaLine, ',')) + matchCount++; + } + }); + + Assert.Equal(8_192, matchCount); + Assert.True( + allocatedBytes < 1_024, + $"Trimmed suffix checks allocated {allocatedBytes:N0} bytes"); + } + #if NET8_0 [Fact] #else From 3f41d838c2bd7e266fa1f4b18d7b44a335ef2f38 Mon Sep 17 00:00:00 2001 From: Widthdom Date: Sun, 26 Jul 2026 17:36:38 +0900 Subject: [PATCH 11/30] Allocate functional exclusion spans lazily --- DEVELOPER_GUIDE.md | 6 ++ TESTING_GUIDE.md | 2 + .../+lazy-functional-spans.fixed.md | 23 +++++++ ...tionalLanguageReferenceExtractor.Erlang.cs | 11 ++-- ...ctionalLanguageReferenceExtractor.Ocaml.cs | 11 ++-- ...nctionalLanguageReferenceExtractor.Raku.cs | 8 ++- .../FunctionalLanguageReferenceExtractor.cs | 15 ++++- tests/CodeIndex.Tests/PerformanceTests.cs | 63 +++++++++++++++++++ 8 files changed, 124 insertions(+), 15 deletions(-) create mode 100644 changelog.d/unreleased/+lazy-functional-spans.fixed.md diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index 223291ad61..3bdd293174 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -529,6 +529,9 @@ line without materializing padded copies. `SpanCharacterSearch.EndsWithAfterTrim` is the shared suffix primitive for these sentinels and for CSS selector continuations plus C# / Java body-less declaration termination. +Functional-language exclusion span lists are lazy. Erlang quoted/remote calls, +OCAML type/qualified calls, and Raku qualified/method calls must not allocate an +empty list on every source line when no corresponding match exists. All line-based symbol and reference extractors share `SourceLineSplitter`. It counts newline boundaries once, allocates the exact result array, and then materializes only the line strings that downstream scanners require; do not @@ -3548,6 +3551,9 @@ state-machine の sentinel 判定も span 上で行う。Erlang specification / と Raku heredoc terminator は、padding を含む copy を実体化せず original line の view を trim する。 `SpanCharacterSearch.EndsWithAfterTrim` はこれらの sentinel に加え、CSS selector continuation と C# / Java の body-less declaration termination が共有する suffix primitive である。 +functional-language の exclusion span list は lazy にする。Erlang quoted / remote call、 +OCAML type / qualified call、Raku qualified / method call は、対応する match がない source line +ごとに empty list を割り当ててはならない。 line-based symbol / reference extractor はすべて `SourceLineSplitter` を共有する。 newline boundary を一度数えて exact result array を確保し、downstream scanner が必要とする line string だけを実体化する。`string.Split` による separator-index array を戻してはならない。 diff --git a/TESTING_GUIDE.md b/TESTING_GUIDE.md index 1003290ca3..7d7dba5dcc 100644 --- a/TESTING_GUIDE.md +++ b/TESTING_GUIDE.md @@ -585,6 +585,7 @@ Use `docs/test-doc-maintenance-plan.md` before moving oversized suites or adding `SourceLineSplitting_LargeFiles_AvoidsSeparatorIndexArrays` covers the shared all-language symbol/reference line splitter with 8,192 lines and fixes its exact output plus bounded allocation contract. `FunctionalTerminatorChecks_LongPaddedLines_DoNotAllocate` keeps Erlang specification/callable and Raku heredoc state-machine sentinels on trimmed spans rather than padded string copies. `TrimmedSuffixChecks_LongDeclarationLines_DoNotAllocate` keeps CSS selector continuations and C# / Java body-less declaration suffix checks allocation-free on long padded lines. + `FunctionalReferenceExtraction_CallFreeLines_AvoidsEmptySpanLists` covers 4,096 call-free lines each of Erlang, OCAML, and Raku so their exclusion span lists remain lazy. `ReferenceExtraction_CSharpNoAliasDenseReferences_StaysWithinAllocationBudget` keeps 12,000 no-alias calls from paying for alias dedupe, `CSharpAliasCompaction_DenseDuplicates_StaysWithinAllocationBudget` guards pre-sized stable compaction after alias rewrites, and `MutualRecursion_DenseRepeatedQualifiedNames_StaysWithinAllocationBudget` covers repeated qualified C#- and Python-style cycle names without per-edge normalization strings. `ReusableStatSnapshot_OnePassMaterialization_StaysWithinAllocationBudget` warms a 1,024-row typed snapshot with its exact capacity, then prevents a second candidate collection or redundant path values from returning to the default `net8.0` path. `ReferenceBatchTransactions_RepositoryScaleAtomicFileScopeEliminatesControlledSqlScopes` models 321,352 references distributed over 856 files in the repository snapshot's five/six-batch shape and fixes the control-SQL contract at 5,009 public batch transaction scopes versus zero explicit atomic-file batch scopes; keep it deterministic and allocation-light instead of inserting all modeled rows or asserting elapsed time. For reference-line window performance audits, measure identical prebuilt reference rows against one-batch and 32-batch limits in alternating order, report both elapsed time and `GC.GetAllocatedBytesForCurrentThread`, and remove the timing harness after recording the result; end-to-end `--memory-trace` rebuilds remain corroborating evidence because extraction, graph finalization, and OS page-cache variance can dominate the persistence delta. @@ -1443,6 +1444,7 @@ dotnet test --filter "FullyQualifiedName~GitHelperTests" `SourceLineSplitting_LargeFiles_AvoidsSeparatorIndexArrays` は、全言語の symbol / reference extraction が共有する line splitter を8,192行で検証し、exact output と bounded allocation contract を固定します。 `FunctionalTerminatorChecks_LongPaddedLines_DoNotAllocate` は、Erlang specification / callable と Raku heredoc の state-machine sentinel を、padding 付き string copy ではなく trimmed span 上に固定します。 `TrimmedSuffixChecks_LongDeclarationLines_DoNotAllocate` は、長い padding 付き行に対する CSS selector continuation と C# / Java body-less declaration の suffix 判定を allocation-free に固定します。 + `FunctionalReferenceExtraction_CallFreeLines_AvoidsEmptySpanLists` は、Erlang、OCAML、Raku それぞれ4,096行の call-free input を使い、exclusion span list が lazy なままであることを固定します。 `ReferenceExtraction_CSharpNoAliasDenseReferences_StaysWithinAllocationBudget` は alias のない 12,000 call が alias dedupe のコストを負わないこと、`CSharpAliasCompaction_DenseDuplicates_StaysWithinAllocationBudget` は alias rewrite 後の事前 capacity 付き stable compaction、`MutualRecursion_DenseRepeatedQualifiedNames_StaysWithinAllocationBudget` は C# / Python 形式の qualified cycle name が edge ごとの正規化文字列を作らないことを保証します。 `ReusableStatSnapshot_OnePassMaterialization_StaysWithinAllocationBudget` は exact capacity を渡した1,024行の typed snapshot を warm-up し、2つ目の候補 collection や重複 path value が通常の `net8.0` 経路へ戻らないようにします。 `ReferenceBatchTransactions_RepositoryScaleAtomicFileScopeEliminatesControlledSqlScopes` は321,352 refsを自己snapshotの5/6 batch形状で856 filesへ分配し、制御SQL契約をpublic batch transaction scope 5,009回対explicit atomic-file scope 0回に固定します。全model rowを挿入したり経過時間をassertしたりせず、deterministicでallocation-lightなまま維持してください。reference-line windowの性能監査では、同一の事前構築済みreference rowを1-batch上限と32-batch上限で交互に測り、経過時間と`GC.GetAllocatedBytesForCurrentThread`の両方を報告して、結果記録後にtiming harnessを削除します。end-to-endの`--memory-trace` rebuildは、extraction・graph finalize・OS page cacheの変動が永続化差を支配し得るため、補助証拠として扱ってください。 diff --git a/changelog.d/unreleased/+lazy-functional-spans.fixed.md b/changelog.d/unreleased/+lazy-functional-spans.fixed.md new file mode 100644 index 0000000000..80e6451d72 --- /dev/null +++ b/changelog.d/unreleased/+lazy-functional-spans.fixed.md @@ -0,0 +1,23 @@ +--- +category: fixed +affected: + - src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.cs + - src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Erlang.cs + - src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Ocaml.cs + - src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Raku.cs + - tests/CodeIndex.Tests/PerformanceTests.cs + - DEVELOPER_GUIDE.md + - TESTING_GUIDE.md +--- + +## English + +- **Stopped allocating empty exclusion lists on functional-language lines** — + Erlang, OCAML, and Raku now create quoted, remote, type, qualified, and method + span lists only after the first relevant match on a line. + +## 日本語 + +- **functional-language line ごとの empty exclusion list allocation を解消しました** — + Erlang、OCAML、Raku は quoted、remote、type、qualified、method span list を、行内の + 最初の関連 match が見つかった後にだけ作ります。 diff --git a/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Erlang.cs b/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Erlang.cs index 85c552da71..2c3978d6f7 100644 --- a/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Erlang.cs +++ b/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Erlang.cs @@ -29,12 +29,13 @@ private static void EmitErlangReferences( } var quotedAtomSpans = GetErlangQuotedAtomSpans(line); - var remoteCallSpans = new List<(int Start, int End)>(); + List<(int Start, int End)>? remoteCallSpans = null; foreach (Match match in ErlangRemoteCallRegex.Matches(line)) { if (IsInsideQuotedAtom(match.Index)) continue; - remoteCallSpans.Add((match.Index, match.Index + match.Length)); + (remoteCallSpans ??= []).Add( + (match.Index, match.Index + match.Length)); AddFunctionalReference(references, seen, fileId, match.Groups["module"], "reference", context, lineNumber, container, "erlang"); AddFunctionalReference(references, seen, fileId, match.Groups["name"], "call", context, lineNumber, container, "erlang"); } @@ -70,9 +71,9 @@ bool IsInsideQuotedAtom(int index) => ContainsFunctionalSpanInterior(quotedAtomSpans, index); } - private static List<(int Start, int End)> GetErlangQuotedAtomSpans(string line) + private static List<(int Start, int End)>? GetErlangQuotedAtomSpans(string line) { - var spans = new List<(int Start, int End)>(); + List<(int Start, int End)>? spans = null; for (var index = 0; index < line.Length; index++) { if (line[index] != '\'') @@ -90,7 +91,7 @@ bool IsInsideQuotedAtom(int index) if (line[index] != '\'') continue; - spans.Add((start, index + 1)); + (spans ??= []).Add((start, index + 1)); break; } } diff --git a/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Ocaml.cs b/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Ocaml.cs index 09db3a92a3..a99464a52c 100644 --- a/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Ocaml.cs +++ b/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Ocaml.cs @@ -41,7 +41,7 @@ private static void EmitOcamlReferences( state.OcamlActiveTypeDefinition = typeDefinition; } - var typeReferenceSpans = new List<(int Start, int End)>(); + List<(int Start, int End)>? typeReferenceSpans = null; AddMatch(OcamlImportRegex.Match(line), "import"); AddMatch(OcamlModuleAliasRegex.Match(line), "alias"); var typeAliasTarget = OcamlTypeAliasTargetRegex.Match(line); @@ -63,7 +63,8 @@ private static void EmitOcamlReferences( return; } - var qualifiedCallSpans = new List<(int Start, int End)>(typeReferenceSpans); + List<(int Start, int End)>? qualifiedCallSpans = + typeReferenceSpans is null ? null : new(typeReferenceSpans); foreach (Match match in OcamlQualifiedCallRegex.Matches(line)) { if (OverlapsFunctionalSpan( @@ -71,7 +72,8 @@ private static void EmitOcamlReferences( match.Index, match.Index + match.Length)) continue; - qualifiedCallSpans.Add((match.Index, match.Index + match.Length)); + (qualifiedCallSpans ??= []).Add( + (match.Index, match.Index + match.Length)); AddFunctionalReference(references, seen, fileId, match.Groups["module"], "reference", context, lineNumber, container, "ocaml"); AddFunctionalReference(references, seen, fileId, match.Groups["name"], "call", context, lineNumber, container, "ocaml"); } @@ -108,7 +110,8 @@ void AddOcamlTypeReference(Group group) if (!group.Success) return; - typeReferenceSpans.Add((group.Index, group.Index + group.Length)); + (typeReferenceSpans ??= []).Add( + (group.Index, group.Index + group.Length)); if (!OcamlIgnoredTypeReferences.Contains(group.Value)) { AddFunctionalReference( diff --git a/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Raku.cs b/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Raku.cs index 20f453fd49..02ff033d0d 100644 --- a/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Raku.cs +++ b/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Raku.cs @@ -36,16 +36,18 @@ private static void EmitRakuReferences( if (typeDefinition != null) return; - var qualifiedCallSpans = new List<(int Start, int End)>(); + List<(int Start, int End)>? qualifiedCallSpans = null; foreach (Match match in RakuQualifiedCallRegex.Matches(line)) { - qualifiedCallSpans.Add((match.Index, match.Index + match.Length)); + (qualifiedCallSpans ??= []).Add( + (match.Index, match.Index + match.Length)); AddFunctionalReference(references, seen, fileId, match.Groups["module"], "reference", context, lineNumber, container, "raku"); AddFunctionalReference(references, seen, fileId, match.Groups["name"], "call", context, lineNumber, container, "raku"); } foreach (Match match in RakuMethodCallRegex.Matches(line)) { - qualifiedCallSpans.Add((match.Index, match.Index + match.Length)); + (qualifiedCallSpans ??= []).Add( + (match.Index, match.Index + match.Length)); AddFunctionalReference(references, seen, fileId, match.Groups["name"], "call", context, lineNumber, container, "raku"); } diff --git a/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.cs b/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.cs index d08e8faece..1a2c355c3a 100644 --- a/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.cs +++ b/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.cs @@ -139,9 +139,12 @@ private sealed class FunctionalReferenceState } internal static bool ContainsFunctionalSpan( - IReadOnlyList<(int Start, int End)> spans, + IReadOnlyList<(int Start, int End)>? spans, int index) { + if (spans is null) + return false; + for (var spanIndex = 0; spanIndex < spans.Count; spanIndex++) { var span = spans[spanIndex]; @@ -153,9 +156,12 @@ internal static bool ContainsFunctionalSpan( } private static bool ContainsFunctionalSpanInterior( - IReadOnlyList<(int Start, int End)> spans, + IReadOnlyList<(int Start, int End)>? spans, int index) { + if (spans is null) + return false; + for (var spanIndex = 0; spanIndex < spans.Count; spanIndex++) { var span = spans[spanIndex]; @@ -167,10 +173,13 @@ private static bool ContainsFunctionalSpanInterior( } internal static bool OverlapsFunctionalSpan( - IReadOnlyList<(int Start, int End)> spans, + IReadOnlyList<(int Start, int End)>? spans, int start, int end) { + if (spans is null) + return false; + for (var spanIndex = 0; spanIndex < spans.Count; spanIndex++) { var span = spans[spanIndex]; diff --git a/tests/CodeIndex.Tests/PerformanceTests.cs b/tests/CodeIndex.Tests/PerformanceTests.cs index 971a28801c..f921069d96 100644 --- a/tests/CodeIndex.Tests/PerformanceTests.cs +++ b/tests/CodeIndex.Tests/PerformanceTests.cs @@ -694,6 +694,69 @@ public void TrimmedSuffixChecks_LongDeclarationLines_DoNotAllocate() $"Trimmed suffix checks allocated {allocatedBytes:N0} bytes"); } +#if NET8_0 + [Fact] +#else + [Fact(Skip = PracticalBudgetTestTarget.SecondaryTargetSkipReason)] +#endif + public void FunctionalReferenceExtraction_CallFreeLines_AvoidsEmptySpanLists() + { + const int lineCount = 4_096; + var fixtures = new[] + { + ( + Language: "erlang", + Content: string.Join( + '\n', + Enumerable.Range(0, lineCount) + .Select(index => $"value_{index} = {index}."))), + ( + Language: "ocaml", + Content: string.Join( + '\n', + Enumerable.Range(0, lineCount) + .Select(index => $"let value_{index} = {index}"))), + ( + Language: "raku", + Content: string.Join( + '\n', + Enumerable.Range(0, lineCount) + .Select(index => $"my $value_{index} = {index};"))), + } + .Select(fixture => ( + fixture.Language, + fixture.Content, + Symbols: SymbolExtractor.Extract( + 1, + fixture.Language, + fixture.Content))) + .ToArray(); + foreach (var fixture in fixtures) + { + _ = ReferenceExtractor.Extract( + 1, + fixture.Language, + fixture.Content, + fixture.Symbols); + } + + var allocatedBytes = MeasureAllocatedBytes(() => + { + foreach (var fixture in fixtures) + { + _ = ReferenceExtractor.Extract( + 1, + fixture.Language, + fixture.Content, + fixture.Symbols); + } + }); + + Assert.True( + allocatedBytes < 14_000_000, + $"Call-free functional extraction allocated {allocatedBytes:N0} bytes"); + } + #if NET8_0 [Fact] #else From 14d88e89418ce1d54a2815441cdf810f3d64dfe8 Mon Sep 17 00:00:00 2001 From: Widthdom Date: Sun, 26 Jul 2026 17:41:07 +0900 Subject: [PATCH 12/30] Stream bounded regex match enumeration --- DEVELOPER_GUIDE.md | 4 +- TESTING_GUIDE.md | 4 + ...+stream-bounded-regex-enumeration.fixed.md | 19 +++++ src/CodeIndex/Indexer/BoundedRegex.cs | 82 +++++++++++++------ tests/CodeIndex.Tests/BoundedRegexTests.cs | 26 ++++++ 5 files changed, 109 insertions(+), 26 deletions(-) create mode 100644 changelog.d/unreleased/+stream-bounded-regex-enumeration.fixed.md diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index 3bdd293174..d23dfed778 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -2847,7 +2847,7 @@ Contract guarantees: - **Indexing and configured extractor patterns.** Indexing diagnostics use `regex_timeout`; configured pattern diagnostics use `pattern_regex_timeout`. Indexing skips the affected file or pattern so the run can finish and reports bounded diagnostics instead of leaking the pathological pattern input. - **Query/find and MCP find.** CLI human/JSON errors and MCP error envelopes use `regex_timeout` with the same timeout duration text. The recovery hint is surface-specific only where CLI flags and MCP tool arguments differ. - **Redaction surfaces fail closed.** `DiagnosticRedactor`, `GlobalToolLog`, and MCP audit argument values replace the affected value with the configured redaction placeholder. Sensitive-name decisions use `SensitiveNameClassifier`, which normalizes separators and case before checking shared credential fragments so diagnostic and audit redaction cannot drift. `DiagnosticSanitizer` omits the whole message with `[message omitted after sanitization timeout]`. `SuggestionStore` records `redaction_timeout` and persists `[REDACTED:redaction_timeout]`. GitHub API response bodies are replaced with `[response body omitted after redaction timeout]`. -- **Bounded extraction helpers.** `BoundedRegex` keeps extraction best-effort by returning empty matches/`false` or the original input depending on the operation, and records captured timeout diagnostics when a capture scope is active. +- **Bounded extraction helpers.** `BoundedRegex` keeps extraction best-effort by returning empty matches/`false` or the original input depending on the operation, and records captured timeout diagnostics when a capture scope is active. `EnumerateMatches` advances with `Match.NextMatch` only when the consumer requests another result, so bounded extractor loops can stop without materializing the rest of a dense match collection. ## Metrics emission @@ -5339,7 +5339,7 @@ Regex timeout の挙動は `RegexTimeoutPolicy` (`src/CodeIndex/Diagnostics/Rege - **indexing と configured extractor pattern。** indexing 診断は `regex_timeout`、configured pattern 診断は `pattern_regex_timeout` を使う。実行を完了できるよう、影響を受けたファイルまたは pattern を skip し、病的な pattern 入力を漏らさず bounded diagnostics を報告する。 - **query/find と MCP find。** CLI の human/JSON エラーと MCP error envelope は、同じ timeout duration 表記で `regex_timeout` を使う。CLI flag と MCP tool argument が異なる箇所だけ、復旧 hint を surface 別にする。 - **redaction surface は fail closed。** `DiagnosticRedactor`、`GlobalToolLog`、MCP audit の argument value は、対象値を設定済み redaction placeholder へ置換する。sensitive name 判定は `SensitiveNameClassifier` を使い、区切り文字と大小文字を正規化して共有 credential fragment を確認するため、diagnostic と audit の redaction がずれない。`DiagnosticSanitizer` は `[message omitted after sanitization timeout]` でメッセージ全体を省略する。`SuggestionStore` は `redaction_timeout` を記録し `[REDACTED:redaction_timeout]` を永続化する。GitHub API response body は `[response body omitted after redaction timeout]` に置換する。 -- **bounded extraction helper。** `BoundedRegex` は extraction を best-effort に保つため、operation に応じて empty matches / `false` / 元入力を返し、capture scope が有効な場合は timeout diagnostics を記録する。 +- **bounded extraction helper。** `BoundedRegex` は extraction を best-effort に保つため、operation に応じて empty matches / `false` / 元入力を返し、capture scope が有効な場合は timeout diagnostics を記録する。`EnumerateMatches` は consumer が次の結果を要求したときだけ `Match.NextMatch` で進むため、bounded extractor loop は dense match collection の残りを実体化せず停止できる。 ## メトリクス出力 diff --git a/TESTING_GUIDE.md b/TESTING_GUIDE.md index 7d7dba5dcc..96ec672b20 100644 --- a/TESTING_GUIDE.md +++ b/TESTING_GUIDE.md @@ -318,6 +318,8 @@ Use `docs/test-doc-maintenance-plan.md` before moving oversized suites or adding CI and release workflow contract tests. Keep repeated related workflow/script string contract assertions, including test-result artifact, retry-output, install, Homebrew, changelog, release-payload job splits, container image, SBOM, NuGet publish, secret scope, SDK pin, tool/action pin, and runner/cache policy contracts, in small grouped helpers so the tests emphasize the contract being checked; use the comparison-aware helpers when the contract intentionally requires ordinal matching. Release workflow package-normalization ZIP fixture helpers live in `ReleaseWorkflowTests.PackageHelpers.cs` so workflow assertions stay near the workflow contracts. - `PackageNormalizeDiagnosticsTests.cs` Package normalizer diagnostic redaction coverage. Keep timeout-budget assertions aligned with the shared diagnostic redaction policy so high-load full-suite runs do not treat expected path/secret placeholders as flaky. +- `BoundedRegexTests.cs` + Shared extractor-regex safety coverage. Demand-driven enumeration tests must take only the requested prefix and keep a catastrophic suffix unvisited, while timeout tests continue to verify best-effort empty results and captured diagnostics. - `DocumentationStatusContractTests.cs`, `DocumentationDriftTests.cs` Checked-in documentation contract tests. They use `RepositoryTestPaths` to keep status fields, workflow references, documented `cdidx` command examples, release/changelog workflow snippets, and representative English/Japanese guide sections synchronized. `DocumentationStatusContractTests.cs` includes readiness, maintenance, and MCP status fields so status JSON support contracts stay visible in the user and agent guides. @@ -1186,6 +1188,8 @@ dotnet test --filter "FullyQualifiedName~GitHelperTests" CI と release workflow の契約テスト。test-result artifact、retry-output、install、Homebrew、changelog、release-payload job split、container image、SBOM、NuGet publish、secret scope、SDK pin、tool/action pin、runner/cache policy の契約も含め、繰り返しの関連する workflow/script string contract assertion は小さな grouped helper に寄せ、テスト本文が確認している契約を読み取りやすくしてください。contract が ordinal matching を明示的に必要とする場合は comparison-aware helper を使います。Release workflow の package-normalization ZIP fixture helper は `ReleaseWorkflowTests.PackageHelpers.cs` に置き、workflow assertion が workflow 契約の近くに残るようにします。 - `PackageNormalizeDiagnosticsTests.cs` package normalizer の diagnostic redaction カバレッジです。高負荷の full-suite 実行で、期待される path / secret placeholder が flaky に見えないよう、timeout budget の assertion は共有 diagnostic redaction policy と同期させてください。 +- `BoundedRegexTests.cs` + 共有 extractor-regex safety のカバレッジです。demand-driven enumeration test は要求された prefix だけを取得して catastrophic suffix を未走査のまま保ち、timeout test は引き続き best-effort の empty result と captured diagnostic を検証してください。 - `DocumentationStatusContractTests.cs`、`DocumentationDriftTests.cs` checked-in documentation の契約テスト。`RepositoryTestPaths` を使って、status field、workflow 参照、文書化された `cdidx` コマンド例、release/changelog workflow の snippet、代表的な英日 guide セクションの同期を維持します。 `DocumentationStatusContractTests.cs` は readiness、maintenance、MCP status field も含め、status JSON support contract が user guide と agent guide に残るようにします。 diff --git a/changelog.d/unreleased/+stream-bounded-regex-enumeration.fixed.md b/changelog.d/unreleased/+stream-bounded-regex-enumeration.fixed.md new file mode 100644 index 0000000000..fca40dc863 --- /dev/null +++ b/changelog.d/unreleased/+stream-bounded-regex-enumeration.fixed.md @@ -0,0 +1,19 @@ +--- +category: fixed +affected: + - src/CodeIndex/Indexer/BoundedRegex.cs + - tests/CodeIndex.Tests/BoundedRegexTests.cs + - DEVELOPER_GUIDE.md + - TESTING_GUIDE.md +--- + +## English + +- **Made bounded regex enumeration demand-driven** — Extractors that stop after + reaching a result limit no longer force every remaining regex match on the + source line to be materialized first. + +## 日本語 + +- **bounded regex enumeration を demand-driven にしました** — 結果上限に達して + 停止する extractor は、source line に残るすべての regex match を先に実体化しなくなりました。 diff --git a/src/CodeIndex/Indexer/BoundedRegex.cs b/src/CodeIndex/Indexer/BoundedRegex.cs index 49a17252a6..9d024e00f2 100644 --- a/src/CodeIndex/Indexer/BoundedRegex.cs +++ b/src/CodeIndex/Indexer/BoundedRegex.cs @@ -145,20 +145,13 @@ public static BclMatch Match(BclRegex regex, string input) public static IEnumerable EnumerateMatches(BclRegex regex, string input) { - MatchCollection matches; - try + var pattern = regex.ToString(); + var match = FirstMatchOrEmpty(regex, input, pattern); + while (match.Success) { - matches = regex.Matches(input); - _ = matches.Count; - } - catch (RegexMatchTimeoutException ex) - { - RecordTimeout("matches", regex.ToString(), ex); - yield break; - } - - foreach (BclMatch match in matches) yield return match; + match = NextMatchOrEmpty(match, pattern); + } } public static IEnumerable EnumerateMatches(string input, string pattern) => @@ -166,20 +159,12 @@ public static IEnumerable EnumerateMatches(string input, string patter public static IEnumerable EnumerateMatches(string input, string pattern, RegexOptions options) { - MatchCollection matches; - try - { - matches = BclRegex.Matches(input, pattern, options, DefaultMatchTimeout); - _ = matches.Count; - } - catch (RegexMatchTimeoutException ex) + var match = FirstMatchOrEmpty(input, pattern, options); + while (match.Success) { - RecordTimeout("matches", pattern, ex); - yield break; - } - - foreach (BclMatch match in matches) yield return match; + match = NextMatchOrEmpty(match, pattern); + } } public static new bool IsMatch(string input, string pattern) => @@ -354,6 +339,55 @@ public static IEnumerable EnumerateMatches(string input, string patter private static void RecordTimeout(string operation, string pattern, RegexMatchTimeoutException ex) => TimeoutCaptureScope.Value?.Record(operation, pattern, ex.MatchTimeout); + private static BclMatch FirstMatchOrEmpty( + BclRegex regex, + string input, + string pattern) + { + try + { + return regex.Match(input); + } + catch (RegexMatchTimeoutException ex) + { + RecordTimeout("matches", pattern, ex); + return BclMatch.Empty; + } + } + + private static BclMatch FirstMatchOrEmpty( + string input, + string pattern, + RegexOptions options) + { + try + { + return BclRegex.Match( + input, + pattern, + options, + DefaultMatchTimeout); + } + catch (RegexMatchTimeoutException ex) + { + RecordTimeout("matches", pattern, ex); + return BclMatch.Empty; + } + } + + private static BclMatch NextMatchOrEmpty(BclMatch match, string pattern) + { + try + { + return match.NextMatch(); + } + catch (RegexMatchTimeoutException ex) + { + RecordTimeout("matches", pattern, ex); + return BclMatch.Empty; + } + } + private static string HashPattern(string pattern) { var hash = SHA256.HashData(Encoding.UTF8.GetBytes(pattern)); diff --git a/tests/CodeIndex.Tests/BoundedRegexTests.cs b/tests/CodeIndex.Tests/BoundedRegexTests.cs index 8094cfc4c0..ea2786e31a 100644 --- a/tests/CodeIndex.Tests/BoundedRegexTests.cs +++ b/tests/CodeIndex.Tests/BoundedRegexTests.cs @@ -42,6 +42,32 @@ public void EnumerateMatches_InstanceRegexTimeout_ReturnsEmpty() Assert.Empty(matches); } + [Fact] + public void EnumerateMatches_InstanceRegex_StopsAfterConsumerBreak() + { + var regex = new BoundedRegex( + @"token|(?:a+)+$", + default, + TimeSpan.FromMilliseconds(1)); + var input = "token " + new string('a', 10_000) + "!"; + + var match = BoundedRegex.EnumerateMatches(regex, input).Take(1).Single(); + + Assert.Equal("token", match.Value); + } + + [Fact] + public void EnumerateMatches_StaticPattern_StreamsInMatchOrder() + { + var matches = BoundedRegex + .EnumerateMatches("alpha beta gamma", @"\w+") + .Take(2) + .Select(match => match.Value) + .ToArray(); + + Assert.Equal(["alpha", "beta"], matches); + } + [Fact] public void InstanceMatch_Timeout_ReturnsEmpty() { From 74b182a39737154b0673a0d8f28772e0c876d258 Mon Sep 17 00:00:00 2001 From: Widthdom Date: Sun, 26 Jul 2026 17:44:26 +0900 Subject: [PATCH 13/30] Stream functional reference matches --- DEVELOPER_GUIDE.md | 6 +++ TESTING_GUIDE.md | 2 + ...ream-functional-reference-matches.fixed.md | 23 +++++++++++ .../Languages/ElixirReferenceExtractor.cs | 13 ++++++- ...ionalLanguageReferenceExtractor.Clojure.cs | 7 +++- ...tionalLanguageReferenceExtractor.Erlang.cs | 14 ++++++- ...ctionalLanguageReferenceExtractor.Ocaml.cs | 23 +++++++++-- ...nctionalLanguageReferenceExtractor.Raku.cs | 39 ++++++++++++++++--- 8 files changed, 114 insertions(+), 13 deletions(-) create mode 100644 changelog.d/unreleased/+stream-functional-reference-matches.fixed.md diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index d23dfed778..2378efcf0c 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -532,6 +532,9 @@ declaration termination. Functional-language exclusion span lists are lazy. Erlang quoted/remote calls, OCAML type/qualified calls, and Raku qualified/method calls must not allocate an empty list on every source line when no corresponding match exists. +Functional-language regex loops enumerate matches on demand and stop as soon as +the bounded reference list is full. Keep this contract across Clojure, Elixir, +Erlang, OCAML, and Raku so one dense line cannot force unused match objects. All line-based symbol and reference extractors share `SourceLineSplitter`. It counts newline boundaries once, allocates the exact result array, and then materializes only the line strings that downstream scanners require; do not @@ -3554,6 +3557,9 @@ state-machine の sentinel 判定も span 上で行う。Erlang specification / functional-language の exclusion span list は lazy にする。Erlang quoted / remote call、 OCAML type / qualified call、Raku qualified / method call は、対応する match がない source line ごとに empty list を割り当ててはならない。 +functional-language の regex loop は match を demand-driven に列挙し、bounded reference list +が満杯になった時点で停止する。Clojure、Elixir、Erlang、OCAML、Raku でこの契約を維持し、 +dense な1行に対して未使用の match object を強制的に作らない。 line-based symbol / reference extractor はすべて `SourceLineSplitter` を共有する。 newline boundary を一度数えて exact result array を確保し、downstream scanner が必要とする line string だけを実体化する。`string.Split` による separator-index array を戻してはならない。 diff --git a/TESTING_GUIDE.md b/TESTING_GUIDE.md index 96ec672b20..1fe3314410 100644 --- a/TESTING_GUIDE.md +++ b/TESTING_GUIDE.md @@ -588,6 +588,7 @@ Use `docs/test-doc-maintenance-plan.md` before moving oversized suites or adding `FunctionalTerminatorChecks_LongPaddedLines_DoNotAllocate` keeps Erlang specification/callable and Raku heredoc state-machine sentinels on trimmed spans rather than padded string copies. `TrimmedSuffixChecks_LongDeclarationLines_DoNotAllocate` keeps CSS selector continuations and C# / Java body-less declaration suffix checks allocation-free on long padded lines. `FunctionalReferenceExtraction_CallFreeLines_AvoidsEmptySpanLists` covers 4,096 call-free lines each of Erlang, OCAML, and Raku so their exclusion span lists remain lazy. + Functional-language graph fixtures also cover the demand-driven Clojure, Elixir, Erlang, OCAML, and Raku regex loops; keep their bounded-list exit checks adjacent to enumeration. `ReferenceExtraction_CSharpNoAliasDenseReferences_StaysWithinAllocationBudget` keeps 12,000 no-alias calls from paying for alias dedupe, `CSharpAliasCompaction_DenseDuplicates_StaysWithinAllocationBudget` guards pre-sized stable compaction after alias rewrites, and `MutualRecursion_DenseRepeatedQualifiedNames_StaysWithinAllocationBudget` covers repeated qualified C#- and Python-style cycle names without per-edge normalization strings. `ReusableStatSnapshot_OnePassMaterialization_StaysWithinAllocationBudget` warms a 1,024-row typed snapshot with its exact capacity, then prevents a second candidate collection or redundant path values from returning to the default `net8.0` path. `ReferenceBatchTransactions_RepositoryScaleAtomicFileScopeEliminatesControlledSqlScopes` models 321,352 references distributed over 856 files in the repository snapshot's five/six-batch shape and fixes the control-SQL contract at 5,009 public batch transaction scopes versus zero explicit atomic-file batch scopes; keep it deterministic and allocation-light instead of inserting all modeled rows or asserting elapsed time. For reference-line window performance audits, measure identical prebuilt reference rows against one-batch and 32-batch limits in alternating order, report both elapsed time and `GC.GetAllocatedBytesForCurrentThread`, and remove the timing harness after recording the result; end-to-end `--memory-trace` rebuilds remain corroborating evidence because extraction, graph finalization, and OS page-cache variance can dominate the persistence delta. @@ -1449,6 +1450,7 @@ dotnet test --filter "FullyQualifiedName~GitHelperTests" `FunctionalTerminatorChecks_LongPaddedLines_DoNotAllocate` は、Erlang specification / callable と Raku heredoc の state-machine sentinel を、padding 付き string copy ではなく trimmed span 上に固定します。 `TrimmedSuffixChecks_LongDeclarationLines_DoNotAllocate` は、長い padding 付き行に対する CSS selector continuation と C# / Java body-less declaration の suffix 判定を allocation-free に固定します。 `FunctionalReferenceExtraction_CallFreeLines_AvoidsEmptySpanLists` は、Erlang、OCAML、Raku それぞれ4,096行の call-free input を使い、exclusion span list が lazy なままであることを固定します。 + functional-language graph fixture は Clojure、Elixir、Erlang、OCAML、Raku の demand-driven regex loop も検証します。bounded-list の終了判定は enumeration の直近に維持してください。 `ReferenceExtraction_CSharpNoAliasDenseReferences_StaysWithinAllocationBudget` は alias のない 12,000 call が alias dedupe のコストを負わないこと、`CSharpAliasCompaction_DenseDuplicates_StaysWithinAllocationBudget` は alias rewrite 後の事前 capacity 付き stable compaction、`MutualRecursion_DenseRepeatedQualifiedNames_StaysWithinAllocationBudget` は C# / Python 形式の qualified cycle name が edge ごとの正規化文字列を作らないことを保証します。 `ReusableStatSnapshot_OnePassMaterialization_StaysWithinAllocationBudget` は exact capacity を渡した1,024行の typed snapshot を warm-up し、2つ目の候補 collection や重複 path value が通常の `net8.0` 経路へ戻らないようにします。 `ReferenceBatchTransactions_RepositoryScaleAtomicFileScopeEliminatesControlledSqlScopes` は321,352 refsを自己snapshotの5/6 batch形状で856 filesへ分配し、制御SQL契約をpublic batch transaction scope 5,009回対explicit atomic-file scope 0回に固定します。全model rowを挿入したり経過時間をassertしたりせず、deterministicでallocation-lightなまま維持してください。reference-line windowの性能監査では、同一の事前構築済みreference rowを1-batch上限と32-batch上限で交互に測り、経過時間と`GC.GetAllocatedBytesForCurrentThread`の両方を報告して、結果記録後にtiming harnessを削除します。end-to-endの`--memory-trace` rebuildは、extraction・graph finalize・OS page cacheの変動が永続化差を支配し得るため、補助証拠として扱ってください。 diff --git a/changelog.d/unreleased/+stream-functional-reference-matches.fixed.md b/changelog.d/unreleased/+stream-functional-reference-matches.fixed.md new file mode 100644 index 0000000000..578d6bac07 --- /dev/null +++ b/changelog.d/unreleased/+stream-functional-reference-matches.fixed.md @@ -0,0 +1,23 @@ +--- +category: fixed +affected: + - src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Clojure.cs + - src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Erlang.cs + - src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Ocaml.cs + - src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Raku.cs + - src/CodeIndex/Indexer/References/Languages/ElixirReferenceExtractor.cs + - DEVELOPER_GUIDE.md + - TESTING_GUIDE.md +--- + +## English + +- **Streamed functional-language reference matches** — Clojure, Elixir, + Erlang, OCAML, and Raku now enumerate regex results on demand and stop dense + lines as soon as the per-file reference budget is full. + +## 日本語 + +- **functional-language の reference match を逐次走査にしました** — Clojure、 + Elixir、Erlang、OCAML、Raku は regex result を demand-driven に列挙し、per-file + reference budget が満杯になると dense line の走査を停止します。 diff --git a/src/CodeIndex/Indexer/References/Languages/ElixirReferenceExtractor.cs b/src/CodeIndex/Indexer/References/Languages/ElixirReferenceExtractor.cs index 0a1432af14..138434f38b 100644 --- a/src/CodeIndex/Indexer/References/Languages/ElixirReferenceExtractor.cs +++ b/src/CodeIndex/Indexer/References/Languages/ElixirReferenceExtractor.cs @@ -76,8 +76,15 @@ private static void EmitDefimplReferences( AddDefimplGroupReference(match.Groups["protocol"]); var typesGroup = match.Groups["types"]; - foreach (Match typeMatch in DefimplTypeRegex.Matches(typesGroup.Value)) + foreach (Match typeMatch in Regex.EnumerateMatches( + DefimplTypeRegex, + typesGroup.Value)) + { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + AddDefimplTypeReference(typeMatch.Groups[0], typesGroup.Index + typeMatch.Index); + } void AddDefimplGroupReference(Group group) => AddDefimplTypeReference(group, group.Index); @@ -120,7 +127,9 @@ public static void EmitAdditionalCallReferences( { if (preparedLine.IndexOf("|>", StringComparison.Ordinal) >= 0) { - foreach (Match match in PipeCallRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + PipeCallRegex, + preparedLine)) { var name = match.Groups["name"].Value; if (!IgnoredPipeCallNames.Contains(name)) diff --git a/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Clojure.cs b/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Clojure.cs index ed83497c39..dd25832b2b 100644 --- a/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Clojure.cs +++ b/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Clojure.cs @@ -85,8 +85,13 @@ private static void EmitClojureReferences( && state.ClojureParenDepth == state.ClojureTypeBodyBaseDepth + 1 ? ClojureCallHeadRegex.Match(callLine) : Match.Empty; - foreach (Match match in ClojureCallHeadRegex.Matches(callLine)) + foreach (Match match in Regex.EnumerateMatches( + ClojureCallHeadRegex, + callLine)) { + if (ReferenceLimitReached(references)) + break; + var fullName = match.Groups["name"].Value; var separator = fullName.LastIndexOf('/'); var name = separator >= 0 ? fullName[(separator + 1)..] : fullName; diff --git a/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Erlang.cs b/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Erlang.cs index 2c3978d6f7..0892995d24 100644 --- a/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Erlang.cs +++ b/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Erlang.cs @@ -30,8 +30,13 @@ private static void EmitErlangReferences( var quotedAtomSpans = GetErlangQuotedAtomSpans(line); List<(int Start, int End)>? remoteCallSpans = null; - foreach (Match match in ErlangRemoteCallRegex.Matches(line)) + foreach (Match match in Regex.EnumerateMatches( + ErlangRemoteCallRegex, + line)) { + if (ReferenceLimitReached(references)) + break; + if (IsInsideQuotedAtom(match.Index)) continue; (remoteCallSpans ??= []).Add( @@ -41,8 +46,13 @@ private static void EmitErlangReferences( } var definitionMatch = ErlangFunctionDefinitionRegex.Match(line); - foreach (Match match in ErlangLocalCallRegex.Matches(line)) + foreach (Match match in Regex.EnumerateMatches( + ErlangLocalCallRegex, + line)) { + if (ReferenceLimitReached(references)) + break; + if (ContainsFunctionalSpan(remoteCallSpans, match.Index)) continue; if (IsInsideQuotedAtom(match.Groups["name"].Index)) diff --git a/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Ocaml.cs b/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Ocaml.cs index a99464a52c..fe89805371 100644 --- a/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Ocaml.cs +++ b/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Ocaml.cs @@ -49,8 +49,15 @@ private static void EmitOcamlReferences( { AddOcamlTypeReference(typeAliasTarget.Groups["name"]); } - foreach (Match match in OcamlTypeReferenceRegex.Matches(line)) + foreach (Match match in Regex.EnumerateMatches( + OcamlTypeReferenceRegex, + line)) + { + if (ReferenceLimitReached(references)) + break; + AddOcamlTypeReference(match.Groups["name"]); + } if (state.OcamlTypeDeclarationMode) return; @@ -65,8 +72,13 @@ private static void EmitOcamlReferences( List<(int Start, int End)>? qualifiedCallSpans = typeReferenceSpans is null ? null : new(typeReferenceSpans); - foreach (Match match in OcamlQualifiedCallRegex.Matches(line)) + foreach (Match match in Regex.EnumerateMatches( + OcamlQualifiedCallRegex, + line)) { + if (ReferenceLimitReached(references)) + break; + if (OverlapsFunctionalSpan( qualifiedCallSpans, match.Index, @@ -79,8 +91,13 @@ private static void EmitOcamlReferences( } var skippedDefinition = false; - foreach (Match match in OcamlBareCallRegex.Matches(line)) + foreach (Match match in Regex.EnumerateMatches( + OcamlBareCallRegex, + line)) { + if (ReferenceLimitReached(references)) + break; + if (ContainsFunctionalSpan(qualifiedCallSpans, match.Index)) continue; diff --git a/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Raku.cs b/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Raku.cs index 02ff033d0d..fe30d70d25 100644 --- a/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Raku.cs +++ b/src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Raku.cs @@ -28,32 +28,61 @@ private static void EmitRakuReferences( if (typeDefinition != null) { - foreach (Match match in RakuTypeRelationRegex.Matches(line)) + foreach (Match match in Regex.EnumerateMatches( + RakuTypeRelationRegex, + line)) + { + if (ReferenceLimitReached(references)) + break; + AddFunctionalReference(references, seen, fileId, match.Groups["name"], "type_reference", context, lineNumber, typeDefinition, "raku"); + } } - foreach (Match match in RakuReturnTypeRegex.Matches(line)) + foreach (Match match in Regex.EnumerateMatches( + RakuReturnTypeRegex, + line)) + { + if (ReferenceLimitReached(references)) + break; + AddFunctionalReference(references, seen, fileId, match.Groups["name"], "type_reference", context, lineNumber, container, "raku"); + } if (typeDefinition != null) return; List<(int Start, int End)>? qualifiedCallSpans = null; - foreach (Match match in RakuQualifiedCallRegex.Matches(line)) + foreach (Match match in Regex.EnumerateMatches( + RakuQualifiedCallRegex, + line)) { + if (ReferenceLimitReached(references)) + break; + (qualifiedCallSpans ??= []).Add( (match.Index, match.Index + match.Length)); AddFunctionalReference(references, seen, fileId, match.Groups["module"], "reference", context, lineNumber, container, "raku"); AddFunctionalReference(references, seen, fileId, match.Groups["name"], "call", context, lineNumber, container, "raku"); } - foreach (Match match in RakuMethodCallRegex.Matches(line)) + foreach (Match match in Regex.EnumerateMatches( + RakuMethodCallRegex, + line)) { + if (ReferenceLimitReached(references)) + break; + (qualifiedCallSpans ??= []).Add( (match.Index, match.Index + match.Length)); AddFunctionalReference(references, seen, fileId, match.Groups["name"], "call", context, lineNumber, container, "raku"); } var skippedDefinition = false; - foreach (Match match in RakuBareCallRegex.Matches(line)) + foreach (Match match in Regex.EnumerateMatches( + RakuBareCallRegex, + line)) { + if (ReferenceLimitReached(references)) + break; + if (ContainsFunctionalSpan(qualifiedCallSpans, match.Index)) continue; From b2ce2f27c6ca03062cd4126eddfce24cb0e3fb01 Mon Sep 17 00:00:00 2001 From: Widthdom Date: Sun, 26 Jul 2026 17:47:26 +0900 Subject: [PATCH 14/30] Stream JVM reference matches --- DEVELOPER_GUIDE.md | 6 ++++ TESTING_GUIDE.md | 2 ++ .../+stream-jvm-reference-matches.fixed.md | 24 ++++++++++++++ .../Languages/GradleReferenceExtractor.cs | 8 +++-- .../JavaReferenceExtractor.Modules.cs | 13 +++++++- .../Languages/JavaReferenceExtractor.Types.cs | 7 ++++- .../Languages/KotlinReferenceExtractor.cs | 31 +++++++++++++++---- .../Languages/ScalaReferenceExtractor.cs | 24 +++++++++++--- .../Support/JvmMethodReferenceExtractor.cs | 7 ++++- 9 files changed, 107 insertions(+), 15 deletions(-) create mode 100644 changelog.d/unreleased/+stream-jvm-reference-matches.fixed.md diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index 2378efcf0c..89a3587a74 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -535,6 +535,9 @@ empty list on every source line when no corresponding match exists. Functional-language regex loops enumerate matches on demand and stop as soon as the bounded reference list is full. Keep this contract across Clojure, Elixir, Erlang, OCAML, and Raku so one dense line cannot force unused match objects. +JVM-family reference scanners follow the same rule. Java, Kotlin, Scala, and +Gradle/Groovy multi-match loops use `BoundedRegex.EnumerateMatches`; loops that +own the bounded reference list stop immediately at its cap. All line-based symbol and reference extractors share `SourceLineSplitter`. It counts newline boundaries once, allocates the exact result array, and then materializes only the line strings that downstream scanners require; do not @@ -3560,6 +3563,9 @@ OCAML type / qualified call、Raku qualified / method call は、対応する ma functional-language の regex loop は match を demand-driven に列挙し、bounded reference list が満杯になった時点で停止する。Clojure、Elixir、Erlang、OCAML、Raku でこの契約を維持し、 dense な1行に対して未使用の match object を強制的に作らない。 +JVM-family reference scanner も同じ規則に従う。Java、Kotlin、Scala、Gradle / Groovy の +multi-match loop は `BoundedRegex.EnumerateMatches` を使い、bounded reference list を +所有する loop は上限に達した時点で停止する。 line-based symbol / reference extractor はすべて `SourceLineSplitter` を共有する。 newline boundary を一度数えて exact result array を確保し、downstream scanner が必要とする line string だけを実体化する。`string.Split` による separator-index array を戻してはならない。 diff --git a/TESTING_GUIDE.md b/TESTING_GUIDE.md index 1fe3314410..26867e350b 100644 --- a/TESTING_GUIDE.md +++ b/TESTING_GUIDE.md @@ -589,6 +589,7 @@ Use `docs/test-doc-maintenance-plan.md` before moving oversized suites or adding `TrimmedSuffixChecks_LongDeclarationLines_DoNotAllocate` keeps CSS selector continuations and C# / Java body-less declaration suffix checks allocation-free on long padded lines. `FunctionalReferenceExtraction_CallFreeLines_AvoidsEmptySpanLists` covers 4,096 call-free lines each of Erlang, OCAML, and Raku so their exclusion span lists remain lazy. Functional-language graph fixtures also cover the demand-driven Clojure, Elixir, Erlang, OCAML, and Raku regex loops; keep their bounded-list exit checks adjacent to enumeration. + JVM graph fixtures cover demand-driven Java type/module/method-reference, Kotlin type/infix/constructor, Scala contextual, and Gradle/Groovy DSL matches. Preserve dense-line ordering when adding cap-aware exits. `ReferenceExtraction_CSharpNoAliasDenseReferences_StaysWithinAllocationBudget` keeps 12,000 no-alias calls from paying for alias dedupe, `CSharpAliasCompaction_DenseDuplicates_StaysWithinAllocationBudget` guards pre-sized stable compaction after alias rewrites, and `MutualRecursion_DenseRepeatedQualifiedNames_StaysWithinAllocationBudget` covers repeated qualified C#- and Python-style cycle names without per-edge normalization strings. `ReusableStatSnapshot_OnePassMaterialization_StaysWithinAllocationBudget` warms a 1,024-row typed snapshot with its exact capacity, then prevents a second candidate collection or redundant path values from returning to the default `net8.0` path. `ReferenceBatchTransactions_RepositoryScaleAtomicFileScopeEliminatesControlledSqlScopes` models 321,352 references distributed over 856 files in the repository snapshot's five/six-batch shape and fixes the control-SQL contract at 5,009 public batch transaction scopes versus zero explicit atomic-file batch scopes; keep it deterministic and allocation-light instead of inserting all modeled rows or asserting elapsed time. For reference-line window performance audits, measure identical prebuilt reference rows against one-batch and 32-batch limits in alternating order, report both elapsed time and `GC.GetAllocatedBytesForCurrentThread`, and remove the timing harness after recording the result; end-to-end `--memory-trace` rebuilds remain corroborating evidence because extraction, graph finalization, and OS page-cache variance can dominate the persistence delta. @@ -1451,6 +1452,7 @@ dotnet test --filter "FullyQualifiedName~GitHelperTests" `TrimmedSuffixChecks_LongDeclarationLines_DoNotAllocate` は、長い padding 付き行に対する CSS selector continuation と C# / Java body-less declaration の suffix 判定を allocation-free に固定します。 `FunctionalReferenceExtraction_CallFreeLines_AvoidsEmptySpanLists` は、Erlang、OCAML、Raku それぞれ4,096行の call-free input を使い、exclusion span list が lazy なままであることを固定します。 functional-language graph fixture は Clojure、Elixir、Erlang、OCAML、Raku の demand-driven regex loop も検証します。bounded-list の終了判定は enumeration の直近に維持してください。 + JVM graph fixture は Java type / module / method-reference、Kotlin type / infix / constructor、Scala contextual、Gradle / Groovy DSL の demand-driven match を検証します。cap-aware exit を追加するときも dense-line の順序を維持してください。 `ReferenceExtraction_CSharpNoAliasDenseReferences_StaysWithinAllocationBudget` は alias のない 12,000 call が alias dedupe のコストを負わないこと、`CSharpAliasCompaction_DenseDuplicates_StaysWithinAllocationBudget` は alias rewrite 後の事前 capacity 付き stable compaction、`MutualRecursion_DenseRepeatedQualifiedNames_StaysWithinAllocationBudget` は C# / Python 形式の qualified cycle name が edge ごとの正規化文字列を作らないことを保証します。 `ReusableStatSnapshot_OnePassMaterialization_StaysWithinAllocationBudget` は exact capacity を渡した1,024行の typed snapshot を warm-up し、2つ目の候補 collection や重複 path value が通常の `net8.0` 経路へ戻らないようにします。 `ReferenceBatchTransactions_RepositoryScaleAtomicFileScopeEliminatesControlledSqlScopes` は321,352 refsを自己snapshotの5/6 batch形状で856 filesへ分配し、制御SQL契約をpublic batch transaction scope 5,009回対explicit atomic-file scope 0回に固定します。全model rowを挿入したり経過時間をassertしたりせず、deterministicでallocation-lightなまま維持してください。reference-line windowの性能監査では、同一の事前構築済みreference rowを1-batch上限と32-batch上限で交互に測り、経過時間と`GC.GetAllocatedBytesForCurrentThread`の両方を報告して、結果記録後にtiming harnessを削除します。end-to-endの`--memory-trace` rebuildは、extraction・graph finalize・OS page cacheの変動が永続化差を支配し得るため、補助証拠として扱ってください。 diff --git a/changelog.d/unreleased/+stream-jvm-reference-matches.fixed.md b/changelog.d/unreleased/+stream-jvm-reference-matches.fixed.md new file mode 100644 index 0000000000..a850856d8a --- /dev/null +++ b/changelog.d/unreleased/+stream-jvm-reference-matches.fixed.md @@ -0,0 +1,24 @@ +--- +category: fixed +affected: + - src/CodeIndex/Indexer/References/Languages/JavaReferenceExtractor.Modules.cs + - src/CodeIndex/Indexer/References/Languages/JavaReferenceExtractor.Types.cs + - src/CodeIndex/Indexer/References/Languages/KotlinReferenceExtractor.cs + - src/CodeIndex/Indexer/References/Languages/ScalaReferenceExtractor.cs + - src/CodeIndex/Indexer/References/Languages/GradleReferenceExtractor.cs + - src/CodeIndex/Indexer/References/Support/JvmMethodReferenceExtractor.cs + - DEVELOPER_GUIDE.md + - TESTING_GUIDE.md +--- + +## English + +- **Streamed JVM-family reference matches** — Java, Kotlin, Scala, and + Gradle/Groovy scanners now consume dense regex results on demand and stop + bounded reference loops once the per-file cap is reached. + +## 日本語 + +- **JVM-family の reference match を逐次走査にしました** — Java、Kotlin、Scala、 + Gradle / Groovy scanner は dense な regex result を demand-driven に消費し、bounded + reference loop は per-file 上限に達すると停止します。 diff --git a/src/CodeIndex/Indexer/References/Languages/GradleReferenceExtractor.cs b/src/CodeIndex/Indexer/References/Languages/GradleReferenceExtractor.cs index 2e78e29494..c9990f8fce 100644 --- a/src/CodeIndex/Indexer/References/Languages/GradleReferenceExtractor.cs +++ b/src/CodeIndex/Indexer/References/Languages/GradleReferenceExtractor.cs @@ -23,14 +23,18 @@ public static void EmitDslCallReferences( { if (preparedLine.IndexOf('{') >= 0) { - foreach (Match match in BlockCallRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + BlockCallRegex, + preparedLine)) addDslReference(match.Groups["name"].Value, match.Groups["name"].Index); } if (!ContainsWhitespace(preparedLine)) return; - foreach (Match match in CommandCallRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + CommandCallRegex, + preparedLine)) addDslReference(match.Groups["name"].Value, match.Groups["name"].Index); } diff --git a/src/CodeIndex/Indexer/References/Languages/JavaReferenceExtractor.Modules.cs b/src/CodeIndex/Indexer/References/Languages/JavaReferenceExtractor.Modules.cs index 2216acc69d..2d4c9de68b 100644 --- a/src/CodeIndex/Indexer/References/Languages/JavaReferenceExtractor.Modules.cs +++ b/src/CodeIndex/Indexer/References/Languages/JavaReferenceExtractor.Modules.cs @@ -33,8 +33,13 @@ public static void EmitDotClassTypeLiteralReferences( int lineNumber, SymbolRecord? container) { - foreach (Match match in DotClassArgRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + DotClassArgRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var argGroup = match.Groups["arg"]; ReferenceExtractor.AddTypeReferenceSegments( references, @@ -80,6 +85,9 @@ public static void EmitModuleDirectiveReferences( foreach (Match match in BoundedRegex.EnumerateMatches(ModuleProvidesDirectiveReferenceRegex, preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var serviceGroup = match.Groups["service"]; ReferenceExtractor.AddTypeReferenceSegment( references, @@ -133,6 +141,9 @@ private static void EmitModuleDirectiveReference( { foreach (Match match in BoundedRegex.EnumerateMatches(regex, preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var nameGroup = match.Groups["name"]; ReferenceExtractor.AddTypeReferenceSegment( references, diff --git a/src/CodeIndex/Indexer/References/Languages/JavaReferenceExtractor.Types.cs b/src/CodeIndex/Indexer/References/Languages/JavaReferenceExtractor.Types.cs index 97cdd83051..81d6d1f074 100644 --- a/src/CodeIndex/Indexer/References/Languages/JavaReferenceExtractor.Types.cs +++ b/src/CodeIndex/Indexer/References/Languages/JavaReferenceExtractor.Types.cs @@ -60,8 +60,13 @@ public static void EmitTypePositionReferences( resolveContainerForColumn, genericParameterNames); - foreach (Match match in InstanceofRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + InstanceofRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var typeGroup = match.Groups["type"]; ReferenceExtractor.AddTypeExpressionSegments( references, diff --git a/src/CodeIndex/Indexer/References/Languages/KotlinReferenceExtractor.cs b/src/CodeIndex/Indexer/References/Languages/KotlinReferenceExtractor.cs index 97c0a5b63c..1a155c52f4 100644 --- a/src/CodeIndex/Indexer/References/Languages/KotlinReferenceExtractor.cs +++ b/src/CodeIndex/Indexer/References/Languages/KotlinReferenceExtractor.cs @@ -197,7 +197,9 @@ public static void EmitInfixCallReferences( IReadOnlySet infixFunctionNames, Action addCallLikeReference) { - foreach (Match match in IdentifierRegex.Matches(originalLine)) + foreach (Match match in Regex.EnumerateMatches( + IdentifierRegex, + originalLine)) { var nameGroup = match.Groups["name"]; var name = nameGroup.Value; @@ -311,8 +313,13 @@ public static void EmitClassLiteralReferences( SymbolRecord? container) { var genericParameterNames = CollectGenericParameterNames(preparedLine); - foreach (Match match in ClassLiteralRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + ClassLiteralRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var typeGroup = match.Groups["type"]; ReferenceExtractor.AddTypeExpressionSegments( references, @@ -341,8 +348,13 @@ public static void EmitBacktickConstructorReferences( if (constructorTypeNames.Count == 0) return; - foreach (Match match in BacktickConstructorCallRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + BacktickConstructorCallRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var nameGroup = match.Groups["name"]; if (IsBacktickConstructorDeclarationSite(preparedLine, nameGroup.Index)) continue; @@ -376,8 +388,10 @@ public static void EmitCtorDelegationReferences( int lineNumber, SymbolRecord? container) { - var matches = CtorDelegationRegex.Matches(preparedLine); - if (matches.Count == 0) + using var matches = Regex + .EnumerateMatches(CtorDelegationRegex, preparedLine) + .GetEnumerator(); + if (!matches.MoveNext()) return; var enclosingType = ReferenceExtractor.FindInnermostClassLike(getEnclosingTypeCandidates(), lineNumber); @@ -392,8 +406,12 @@ public static void EmitCtorDelegationReferences( ctorContainer = FindEnclosingKotlinConstructor(symbols, enclosingType, lineNumber) ?? ctorContainer; } - foreach (Match match in matches) + do { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + + var match = matches.Current; var kindToken = match.Groups["kind"].Value; string? target; if (kindToken == "this") @@ -428,6 +446,7 @@ public static void EmitCtorDelegationReferences( lineNumber, ctorContainer); } + while (matches.MoveNext()); } } diff --git a/src/CodeIndex/Indexer/References/Languages/ScalaReferenceExtractor.cs b/src/CodeIndex/Indexer/References/Languages/ScalaReferenceExtractor.cs index d2fd78c21d..ea1c4f85a8 100644 --- a/src/CodeIndex/Indexer/References/Languages/ScalaReferenceExtractor.cs +++ b/src/CodeIndex/Indexer/References/Languages/ScalaReferenceExtractor.cs @@ -44,7 +44,9 @@ public static void EmitTrailingBlockCallReferences( if (preparedLine.IndexOf('{') < 0) return; - foreach (Match match in TrailingBlockCallRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + TrailingBlockCallRegex, + preparedLine)) { var name = match.Groups["name"].Value; if (IgnoredBlockCallNames.Contains(name)) @@ -66,7 +68,9 @@ public static void EmitAdditionalReferences( { if (preparedLine.IndexOf("<-", StringComparison.Ordinal) >= 0) { - foreach (Match match in ForGeneratorRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + ForGeneratorRegex, + preparedLine)) { var nameGroup = match.Groups["name"]; addCallLikeReference(nameGroup.Value, nameGroup.Index); @@ -88,8 +92,13 @@ public static void EmitAdditionalReferences( if (preparedLine.IndexOf("using", StringComparison.Ordinal) >= 0) { - foreach (Match usingMatch in UsingClauseRegex.Matches(preparedLine)) + foreach (Match usingMatch in Regex.EnumerateMatches( + UsingClauseRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var parameters = usingMatch.Groups["params"]; if (parameters.Value.IndexOf(':') < 0 && parameters.Value.IndexOf('=') < 0) @@ -97,8 +106,15 @@ public static void EmitAdditionalReferences( continue; } - foreach (Match typeMatch in UsingTypeRegex.Matches(parameters.Value)) + foreach (Match typeMatch in Regex.EnumerateMatches( + UsingTypeRegex, + parameters.Value)) + { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + AddTypeReference(typeMatch.Groups["type"], parameters.Index); + } } } diff --git a/src/CodeIndex/Indexer/References/Support/JvmMethodReferenceExtractor.cs b/src/CodeIndex/Indexer/References/Support/JvmMethodReferenceExtractor.cs index f4ebedd22c..2d4b6ec370 100644 --- a/src/CodeIndex/Indexer/References/Support/JvmMethodReferenceExtractor.cs +++ b/src/CodeIndex/Indexer/References/Support/JvmMethodReferenceExtractor.cs @@ -26,8 +26,13 @@ public static void EmitMethodReferenceReferences( int lineNumber, Func resolveContainerForColumn) { - foreach (Match match in MethodReferenceRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + MethodReferenceRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var nameGroup = match.Groups["name"]; var container = resolveContainerForColumn(nameGroup.Index); var ownerGroup = match.Groups["owner"]; From cf9d7eb380d97909185621489dc1c6749f607f72 Mon Sep 17 00:00:00 2001 From: Widthdom Date: Sun, 26 Jul 2026 17:51:39 +0900 Subject: [PATCH 15/30] Stream Python reference matches --- DEVELOPER_GUIDE.md | 8 ++ TESTING_GUIDE.md | 2 + .../+stream-python-reference-matches.fixed.md | 29 ++++++ src/CodeIndex/Indexer/BoundedRegex.cs | 11 ++- .../PythonReferenceExtractor.ClassBases.cs | 28 +++++- ...ythonReferenceExtractor.DataclassFields.cs | 14 ++- ...eferenceExtractor.FrameworkIntegrations.cs | 28 +++++- ...onReferenceExtractor.FunctionSignatures.cs | 52 ++++++++-- .../PythonReferenceExtractor.RuntimeTypes.cs | 98 ++++++++++++++++--- ...ythonReferenceExtractor.TypingFactories.cs | 56 +++++++++-- .../Languages/PythonReferenceExtractor.cs | 29 +++++- tests/CodeIndex.Tests/BoundedRegexTests.cs | 14 +++ 12 files changed, 324 insertions(+), 45 deletions(-) create mode 100644 changelog.d/unreleased/+stream-python-reference-matches.fixed.md diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index 89a3587a74..c1501a5d9a 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -538,6 +538,10 @@ Erlang, OCAML, and Raku so one dense line cannot force unused match objects. JVM-family reference scanners follow the same rule. Java, Kotlin, Scala, and Gradle/Groovy multi-match loops use `BoundedRegex.EnumerateMatches`; loops that own the bounded reference list stop immediately at its cap. +Python reference scanners stream decorators, annotations, runtime type checks, +typing factories, dataclass/attrs integrations, and dynamic imports. The +start-offset overload of `BoundedRegex.EnumerateMatches` keeps decorator +argument scans demand-driven without rescanning the decorator prefix. All line-based symbol and reference extractors share `SourceLineSplitter`. It counts newline boundaries once, allocates the exact result array, and then materializes only the line strings that downstream scanners require; do not @@ -3566,6 +3570,10 @@ dense な1行に対して未使用の match object を強制的に作らない JVM-family reference scanner も同じ規則に従う。Java、Kotlin、Scala、Gradle / Groovy の multi-match loop は `BoundedRegex.EnumerateMatches` を使い、bounded reference list を 所有する loop は上限に達した時点で停止する。 +Python reference scanner は decorator、annotation、runtime type check、typing factory、 +dataclass / attrs integration、dynamic import を逐次走査する。`BoundedRegex.EnumerateMatches` +の start-offset overload により、decorator prefix を再走査せず argument scan も +demand-driven のままにする。 line-based symbol / reference extractor はすべて `SourceLineSplitter` を共有する。 newline boundary を一度数えて exact result array を確保し、downstream scanner が必要とする line string だけを実体化する。`string.Split` による separator-index array を戻してはならない。 diff --git a/TESTING_GUIDE.md b/TESTING_GUIDE.md index 26867e350b..6ca9a9cb99 100644 --- a/TESTING_GUIDE.md +++ b/TESTING_GUIDE.md @@ -590,6 +590,7 @@ Use `docs/test-doc-maintenance-plan.md` before moving oversized suites or adding `FunctionalReferenceExtraction_CallFreeLines_AvoidsEmptySpanLists` covers 4,096 call-free lines each of Erlang, OCAML, and Raku so their exclusion span lists remain lazy. Functional-language graph fixtures also cover the demand-driven Clojure, Elixir, Erlang, OCAML, and Raku regex loops; keep their bounded-list exit checks adjacent to enumeration. JVM graph fixtures cover demand-driven Java type/module/method-reference, Kotlin type/infix/constructor, Scala contextual, and Gradle/Groovy DSL matches. Preserve dense-line ordering when adding cap-aware exits. + Python graph fixtures cover streamed decorator arguments, annotations, runtime type checks, typing factories, dataclass/framework integrations, and dynamic imports. `BoundedRegexTests.EnumerateMatches_InstanceRegex_StartsAtRequestedOffset` fixes the no-prefix-rescan contract used by decorator arguments. `ReferenceExtraction_CSharpNoAliasDenseReferences_StaysWithinAllocationBudget` keeps 12,000 no-alias calls from paying for alias dedupe, `CSharpAliasCompaction_DenseDuplicates_StaysWithinAllocationBudget` guards pre-sized stable compaction after alias rewrites, and `MutualRecursion_DenseRepeatedQualifiedNames_StaysWithinAllocationBudget` covers repeated qualified C#- and Python-style cycle names without per-edge normalization strings. `ReusableStatSnapshot_OnePassMaterialization_StaysWithinAllocationBudget` warms a 1,024-row typed snapshot with its exact capacity, then prevents a second candidate collection or redundant path values from returning to the default `net8.0` path. `ReferenceBatchTransactions_RepositoryScaleAtomicFileScopeEliminatesControlledSqlScopes` models 321,352 references distributed over 856 files in the repository snapshot's five/six-batch shape and fixes the control-SQL contract at 5,009 public batch transaction scopes versus zero explicit atomic-file batch scopes; keep it deterministic and allocation-light instead of inserting all modeled rows or asserting elapsed time. For reference-line window performance audits, measure identical prebuilt reference rows against one-batch and 32-batch limits in alternating order, report both elapsed time and `GC.GetAllocatedBytesForCurrentThread`, and remove the timing harness after recording the result; end-to-end `--memory-trace` rebuilds remain corroborating evidence because extraction, graph finalization, and OS page-cache variance can dominate the persistence delta. @@ -1453,6 +1454,7 @@ dotnet test --filter "FullyQualifiedName~GitHelperTests" `FunctionalReferenceExtraction_CallFreeLines_AvoidsEmptySpanLists` は、Erlang、OCAML、Raku それぞれ4,096行の call-free input を使い、exclusion span list が lazy なままであることを固定します。 functional-language graph fixture は Clojure、Elixir、Erlang、OCAML、Raku の demand-driven regex loop も検証します。bounded-list の終了判定は enumeration の直近に維持してください。 JVM graph fixture は Java type / module / method-reference、Kotlin type / infix / constructor、Scala contextual、Gradle / Groovy DSL の demand-driven match を検証します。cap-aware exit を追加するときも dense-line の順序を維持してください。 + Python graph fixture は decorator argument、annotation、runtime type check、typing factory、dataclass / framework integration、dynamic import の逐次走査を検証します。`BoundedRegexTests.EnumerateMatches_InstanceRegex_StartsAtRequestedOffset` は decorator argument が prefix を再走査しない契約を固定します。 `ReferenceExtraction_CSharpNoAliasDenseReferences_StaysWithinAllocationBudget` は alias のない 12,000 call が alias dedupe のコストを負わないこと、`CSharpAliasCompaction_DenseDuplicates_StaysWithinAllocationBudget` は alias rewrite 後の事前 capacity 付き stable compaction、`MutualRecursion_DenseRepeatedQualifiedNames_StaysWithinAllocationBudget` は C# / Python 形式の qualified cycle name が edge ごとの正規化文字列を作らないことを保証します。 `ReusableStatSnapshot_OnePassMaterialization_StaysWithinAllocationBudget` は exact capacity を渡した1,024行の typed snapshot を warm-up し、2つ目の候補 collection や重複 path value が通常の `net8.0` 経路へ戻らないようにします。 `ReferenceBatchTransactions_RepositoryScaleAtomicFileScopeEliminatesControlledSqlScopes` は321,352 refsを自己snapshotの5/6 batch形状で856 filesへ分配し、制御SQL契約をpublic batch transaction scope 5,009回対explicit atomic-file scope 0回に固定します。全model rowを挿入したり経過時間をassertしたりせず、deterministicでallocation-lightなまま維持してください。reference-line windowの性能監査では、同一の事前構築済みreference rowを1-batch上限と32-batch上限で交互に測り、経過時間と`GC.GetAllocatedBytesForCurrentThread`の両方を報告して、結果記録後にtiming harnessを削除します。end-to-endの`--memory-trace` rebuildは、extraction・graph finalize・OS page cacheの変動が永続化差を支配し得るため、補助証拠として扱ってください。 diff --git a/changelog.d/unreleased/+stream-python-reference-matches.fixed.md b/changelog.d/unreleased/+stream-python-reference-matches.fixed.md new file mode 100644 index 0000000000..9cbee512b3 --- /dev/null +++ b/changelog.d/unreleased/+stream-python-reference-matches.fixed.md @@ -0,0 +1,29 @@ +--- +category: fixed +affected: + - src/CodeIndex/Indexer/BoundedRegex.cs + - src/CodeIndex/Indexer/References/Languages/PythonReferenceExtractor.cs + - src/CodeIndex/Indexer/References/Languages/PythonReferenceExtractor.ClassBases.cs + - src/CodeIndex/Indexer/References/Languages/PythonReferenceExtractor.DataclassFields.cs + - src/CodeIndex/Indexer/References/Languages/PythonReferenceExtractor.FrameworkIntegrations.cs + - src/CodeIndex/Indexer/References/Languages/PythonReferenceExtractor.FunctionSignatures.cs + - src/CodeIndex/Indexer/References/Languages/PythonReferenceExtractor.RuntimeTypes.cs + - src/CodeIndex/Indexer/References/Languages/PythonReferenceExtractor.TypingFactories.cs + - tests/CodeIndex.Tests/BoundedRegexTests.cs + - DEVELOPER_GUIDE.md + - TESTING_GUIDE.md +--- + +## English + +- **Streamed Python reference matches** — Python decorators, annotations, + runtime type checks, typing factories, dataclass/framework integrations, and + dynamic imports now stop producing regex matches at the reference cap; + decorator arguments also stream directly from their start offset. + +## 日本語 + +- **Python の reference match を逐次走査にしました** — decorator、annotation、 + runtime type check、typing factory、dataclass / framework integration、dynamic + import は reference 上限で regex match の生成を停止し、decorator argument も指定 + offset から直接逐次走査します。 diff --git a/src/CodeIndex/Indexer/BoundedRegex.cs b/src/CodeIndex/Indexer/BoundedRegex.cs index 9d024e00f2..8f39ca7a74 100644 --- a/src/CodeIndex/Indexer/BoundedRegex.cs +++ b/src/CodeIndex/Indexer/BoundedRegex.cs @@ -144,9 +144,15 @@ public static BclMatch Match(BclRegex regex, string input) } public static IEnumerable EnumerateMatches(BclRegex regex, string input) + => EnumerateMatches(regex, input, startAt: 0); + + public static IEnumerable EnumerateMatches( + BclRegex regex, + string input, + int startAt) { var pattern = regex.ToString(); - var match = FirstMatchOrEmpty(regex, input, pattern); + var match = FirstMatchOrEmpty(regex, input, startAt, pattern); while (match.Success) { yield return match; @@ -342,11 +348,12 @@ private static void RecordTimeout(string operation, string pattern, RegexMatchTi private static BclMatch FirstMatchOrEmpty( BclRegex regex, string input, + int startAt, string pattern) { try { - return regex.Match(input); + return regex.Match(input, startAt); } catch (RegexMatchTimeoutException ex) { diff --git a/src/CodeIndex/Indexer/References/Languages/PythonReferenceExtractor.ClassBases.cs b/src/CodeIndex/Indexer/References/Languages/PythonReferenceExtractor.ClassBases.cs index 8b076a22b8..be9d22b0e7 100644 --- a/src/CodeIndex/Indexer/References/Languages/PythonReferenceExtractor.ClassBases.cs +++ b/src/CodeIndex/Indexer/References/Languages/PythonReferenceExtractor.ClassBases.cs @@ -25,8 +25,13 @@ public static void EmitClassBaseReferences( if (preparedLine.IndexOf("metaclass", StringComparison.Ordinal) >= 0) { - foreach (Match match in ClassMetaclassTypeRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + ClassMetaclassTypeRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var name = match.Groups["name"].Value; if (isIgnoredName(name)) continue; @@ -47,11 +52,21 @@ public static void EmitClassBaseReferences( if (preparedLine.IndexOf(',') >= 0) { - foreach (Match match in MultipleClassBaseTypesRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + MultipleClassBaseTypesRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var typesGroup = match.Groups["types"]; - foreach (Match typeMatch in TypeNameRegex.Matches(typesGroup.Value)) + foreach (Match typeMatch in Regex.EnumerateMatches( + TypeNameRegex, + typesGroup.Value)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var name = typeMatch.Groups["name"].Value; if (isIgnoredName(name)) continue; @@ -73,8 +88,13 @@ public static void EmitClassBaseReferences( } } - foreach (Match match in SingleClassBaseTypeRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + SingleClassBaseTypeRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var name = match.Groups["name"].Value; if (isIgnoredName(name)) continue; diff --git a/src/CodeIndex/Indexer/References/Languages/PythonReferenceExtractor.DataclassFields.cs b/src/CodeIndex/Indexer/References/Languages/PythonReferenceExtractor.DataclassFields.cs index 1bed43c963..98f4e11882 100644 --- a/src/CodeIndex/Indexer/References/Languages/PythonReferenceExtractor.DataclassFields.cs +++ b/src/CodeIndex/Indexer/References/Languages/PythonReferenceExtractor.DataclassFields.cs @@ -19,8 +19,13 @@ public static void EmitDataclassesFieldsReferences( if (preparedLine.IndexOf("fields", StringComparison.Ordinal) < 0) return; - foreach (Match match in DataclassesFieldsTargetRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + DataclassesFieldsTargetRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var name = match.Groups["name"].Value; if (isIgnoredName(name)) continue; @@ -144,8 +149,13 @@ private static void EmitDataclassFieldDefaultFactoryReferences( return; string? context = null; - foreach (Match match in DataclassFieldDefaultFactoryRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + DataclassFieldDefaultFactoryRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var name = match.Groups["name"].Value; if (isIgnoredName(name)) continue; diff --git a/src/CodeIndex/Indexer/References/Languages/PythonReferenceExtractor.FrameworkIntegrations.cs b/src/CodeIndex/Indexer/References/Languages/PythonReferenceExtractor.FrameworkIntegrations.cs index d18f8deb4c..9b76ee7f72 100644 --- a/src/CodeIndex/Indexer/References/Languages/PythonReferenceExtractor.FrameworkIntegrations.cs +++ b/src/CodeIndex/Indexer/References/Languages/PythonReferenceExtractor.FrameworkIntegrations.cs @@ -20,8 +20,13 @@ public static void EmitAttrsFieldsReferences( || preparedLine.IndexOf("attr", StringComparison.Ordinal) < 0) return; - foreach (Match match in AttrsFieldsTargetRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + AttrsFieldsTargetRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var name = match.Groups["name"].Value; if (isIgnoredName(name)) continue; @@ -53,8 +58,13 @@ public static void EmitPydanticTypeAdapterReferences( || preparedLine.IndexOf("pydantic", StringComparison.Ordinal) < 0) return; - foreach (Match match in PydanticTypeAdapterTargetRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + PydanticTypeAdapterTargetRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var name = match.Groups["name"].Value; if (isIgnoredName(name)) continue; @@ -86,8 +96,13 @@ public static void EmitPytestRaisesReferences( || preparedLine.IndexOf("pytest", StringComparison.Ordinal) < 0) return; - foreach (Match match in PytestRaisesTypeRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + PytestRaisesTypeRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var name = match.Groups["name"].Value; if (isIgnoredName(name)) continue; @@ -119,8 +134,13 @@ public static void EmitContextlibSuppressReferences( || preparedLine.IndexOf("contextlib", StringComparison.Ordinal) < 0) return; - foreach (Match match in ContextlibSuppressTypeRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + ContextlibSuppressTypeRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var name = match.Groups["name"].Value; if (isIgnoredName(name)) continue; diff --git a/src/CodeIndex/Indexer/References/Languages/PythonReferenceExtractor.FunctionSignatures.cs b/src/CodeIndex/Indexer/References/Languages/PythonReferenceExtractor.FunctionSignatures.cs index 30dba09ddb..cf25eba9be 100644 --- a/src/CodeIndex/Indexer/References/Languages/PythonReferenceExtractor.FunctionSignatures.cs +++ b/src/CodeIndex/Indexer/References/Languages/PythonReferenceExtractor.FunctionSignatures.cs @@ -23,8 +23,13 @@ public static void EmitFunctionReturnReferences( return; } - foreach (Match match in FunctionReturnAnnotationExpressionRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + FunctionReturnAnnotationExpressionRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var typeGroup = match.Groups["type"]; EmitPythonTypeExpressionReferences( typeGroup, @@ -38,8 +43,13 @@ public static void EmitFunctionReturnReferences( isIgnoredName); } - foreach (Match match in FunctionReturnTypeRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + FunctionReturnTypeRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var name = match.Groups["name"].Value; if (isIgnoredName(name)) continue; @@ -77,13 +87,26 @@ public static void EmitFunctionParameterReferences( return; } - foreach (Match functionMatch in FunctionParameterListRegex.Matches(preparedLine)) + foreach (Match functionMatch in Regex.EnumerateMatches( + FunctionParameterListRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var paramsGroup = functionMatch.Groups["params"]; foreach (var (parameterSegment, parameterOffset) in EnumeratePythonTopLevelCommaSegments(paramsGroup.Value)) { - foreach (Match annotationMatch in AnnotationExpressionTypeRegex.Matches(parameterSegment)) + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + + foreach (Match annotationMatch in Regex.EnumerateMatches( + AnnotationExpressionTypeRegex, + parameterSegment)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var typeGroup = annotationMatch.Groups["type"]; EmitPythonTypeExpressionReferences( typeGroup, @@ -98,8 +121,13 @@ public static void EmitFunctionParameterReferences( paramsGroup.Index + parameterOffset); } - foreach (Match annotationMatch in DirectAnnotationTypeRegex.Matches(parameterSegment)) + foreach (Match annotationMatch in Regex.EnumerateMatches( + DirectAnnotationTypeRegex, + parameterSegment)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var name = annotationMatch.Groups["name"].Value; if (isIgnoredName(name)) continue; @@ -133,8 +161,13 @@ public static void EmitVariableAnnotationReferences( if (preparedLine.IndexOf(':') < 0) return; - foreach (Match match in VariableAnnotationExpressionRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + VariableAnnotationExpressionRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var typeGroup = match.Groups["type"]; EmitPythonTypeExpressionReferences( typeGroup, @@ -148,8 +181,13 @@ public static void EmitVariableAnnotationReferences( isIgnoredName); } - foreach (Match match in VariableAnnotationTypeRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + VariableAnnotationTypeRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var name = match.Groups["name"].Value; if (isIgnoredName(name)) continue; diff --git a/src/CodeIndex/Indexer/References/Languages/PythonReferenceExtractor.RuntimeTypes.cs b/src/CodeIndex/Indexer/References/Languages/PythonReferenceExtractor.RuntimeTypes.cs index 3411aaa93e..72e521e215 100644 --- a/src/CodeIndex/Indexer/References/Languages/PythonReferenceExtractor.RuntimeTypes.cs +++ b/src/CodeIndex/Indexer/References/Languages/PythonReferenceExtractor.RuntimeTypes.cs @@ -19,8 +19,13 @@ public static void EmitRaiseReferences( if (!StartsWithPythonKeywordStatement(preparedLine, "raise")) return; - foreach (Match match in BareRaiseTypeRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + BareRaiseTypeRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var name = match.Groups["name"].Value; if (isIgnoredName(name)) continue; @@ -53,11 +58,21 @@ public static void EmitExceptReferences( if (preparedLine.IndexOf('(') >= 0) { - foreach (Match match in ExceptTupleTypeRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + ExceptTupleTypeRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var typesGroup = match.Groups["types"]; - foreach (Match typeMatch in TypeNameRegex.Matches(typesGroup.Value)) + foreach (Match typeMatch in Regex.EnumerateMatches( + TypeNameRegex, + typesGroup.Value)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var name = typeMatch.Groups["name"].Value; if (isIgnoredName(name)) continue; @@ -76,8 +91,13 @@ public static void EmitExceptReferences( } } - foreach (Match match in ExceptTypeRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + ExceptTypeRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var name = match.Groups["name"].Value; if (isIgnoredName(name)) continue; @@ -110,11 +130,21 @@ public static void EmitIsInstanceReferences( if (MayContainPythonTupleArgument(preparedLine)) { - foreach (Match match in IsInstanceTupleTypeRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + IsInstanceTupleTypeRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var typesGroup = match.Groups["types"]; - foreach (Match typeMatch in TypeNameRegex.Matches(typesGroup.Value)) + foreach (Match typeMatch in Regex.EnumerateMatches( + TypeNameRegex, + typesGroup.Value)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var name = typeMatch.Groups["name"].Value; if (isIgnoredName(name)) continue; @@ -133,8 +163,13 @@ public static void EmitIsInstanceReferences( } } - foreach (Match match in IsInstanceTypeRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + IsInstanceTypeRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var name = match.Groups["name"].Value; if (isIgnoredName(name)) continue; @@ -167,11 +202,21 @@ public static void EmitIsSubclassReferences( if (MayContainPythonTupleArgument(preparedLine)) { - foreach (Match match in IsSubclassTupleTypeRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + IsSubclassTupleTypeRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var typesGroup = match.Groups["types"]; - foreach (Match typeMatch in TypeNameRegex.Matches(typesGroup.Value)) + foreach (Match typeMatch in Regex.EnumerateMatches( + TypeNameRegex, + typesGroup.Value)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var name = typeMatch.Groups["name"].Value; if (isIgnoredName(name)) continue; @@ -190,8 +235,13 @@ public static void EmitIsSubclassReferences( } } - foreach (Match match in IsSubclassTypeRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + IsSubclassTypeRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var name = match.Groups["name"].Value; if (isIgnoredName(name)) continue; @@ -224,8 +274,13 @@ public static void EmitCastReferences( if (preparedLine.IndexOf("typing", StringComparison.Ordinal) >= 0) { - foreach (Match match in QualifiedCastTypeRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + QualifiedCastTypeRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var name = match.Groups["name"].Value; if (isIgnoredName(name)) continue; @@ -243,8 +298,13 @@ public static void EmitCastReferences( } } - foreach (Match match in CastTypeRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + CastTypeRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var name = match.Groups["name"].Value; if (isIgnoredName(name)) continue; @@ -329,8 +389,13 @@ public static void EmitAssertTypeReferences( if (preparedLine.IndexOf("typing", StringComparison.Ordinal) >= 0) { - foreach (Match match in QualifiedAssertTypeRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + QualifiedAssertTypeRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var name = match.Groups["name"].Value; if (isIgnoredName(name)) continue; @@ -348,8 +413,13 @@ public static void EmitAssertTypeReferences( } } - foreach (Match match in AssertTypeRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + AssertTypeRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var name = match.Groups["name"].Value; if (isIgnoredName(name)) continue; diff --git a/src/CodeIndex/Indexer/References/Languages/PythonReferenceExtractor.TypingFactories.cs b/src/CodeIndex/Indexer/References/Languages/PythonReferenceExtractor.TypingFactories.cs index 1e044e141c..eb5144827e 100644 --- a/src/CodeIndex/Indexer/References/Languages/PythonReferenceExtractor.TypingFactories.cs +++ b/src/CodeIndex/Indexer/References/Languages/PythonReferenceExtractor.TypingFactories.cs @@ -22,8 +22,13 @@ public static void EmitTypeAliasReferences( && !MayStartPythonTypeAliasStatement(preparedLine)) return; - foreach (Match match in TypeAliasRhsExpressionRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + TypeAliasRhsExpressionRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var typeGroup = match.Groups["type"]; EmitPythonTypeExpressionReferences( typeGroup, @@ -66,8 +71,13 @@ public static void EmitNewTypeReferences( if (preparedLine.IndexOf("NewType", StringComparison.Ordinal) < 0) return; - foreach (Match match in NewTypeUnderlyingTypeRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + NewTypeUnderlyingTypeRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var name = match.Groups["name"].Value; if (isIgnoredName(name)) continue; @@ -102,8 +112,13 @@ public static void EmitTypeVarBoundReferences( return; } - foreach (Match match in TypeVarBoundTypeRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + TypeVarBoundTypeRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + EmitPythonTypeExpressionReferences( match.Groups["type"], references, @@ -135,8 +150,13 @@ public static void EmitTypeVarConstraintReferences( if (preparedLine.IndexOf(',') < 0) return; - foreach (Match match in TypeVarConstraintTypesRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + TypeVarConstraintTypesRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var typesGroup = match.Groups["types"]; EmitPythonTypeExpressionReferences( typesGroup, @@ -166,8 +186,13 @@ public static void EmitGetTypeHintsReferences( if (preparedLine.IndexOf("typing", StringComparison.Ordinal) >= 0) { - foreach (Match match in QualifiedGetTypeHintsTargetRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + QualifiedGetTypeHintsTargetRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var name = match.Groups["name"].Value; if (isIgnoredName(name)) continue; @@ -185,8 +210,13 @@ public static void EmitGetTypeHintsReferences( } } - foreach (Match match in GetTypeHintsTargetRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + GetTypeHintsTargetRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var name = match.Groups["name"].Value; if (isIgnoredName(name)) continue; @@ -219,8 +249,13 @@ public static void EmitDynamicImportReferences( if (preparedLine.IndexOf("importlib", StringComparison.Ordinal) >= 0) { - foreach (Match match in ImportlibDynamicImportRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + ImportlibDynamicImportRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + ReferenceExtractor.AddReference( references, seen, @@ -258,8 +293,13 @@ public static void EmitDynamicImportReferences( if (preparedLine.IndexOf("__import__", StringComparison.Ordinal) < 0) return; - foreach (Match match in BuiltinDynamicImportRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + BuiltinDynamicImportRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var literalMatch = BuiltinDynamicImportLiteralRegex.Match(originalLine, match.Index); if (!literalMatch.Success || literalMatch.Index != match.Index) continue; diff --git a/src/CodeIndex/Indexer/References/Languages/PythonReferenceExtractor.cs b/src/CodeIndex/Indexer/References/Languages/PythonReferenceExtractor.cs index add67ceeef..0ff148fd9f 100644 --- a/src/CodeIndex/Indexer/References/Languages/PythonReferenceExtractor.cs +++ b/src/CodeIndex/Indexer/References/Languages/PythonReferenceExtractor.cs @@ -174,8 +174,13 @@ private static void EmitPythonTypeExpressionReferences( { var normalized = NormalizePythonAnnotationExpression(typeGroup.Value); var offsetDelta = typeGroup.Value.Length - normalized.Length; - foreach (Match typeMatch in TypeNameRegex.Matches(normalized)) + foreach (Match typeMatch in Regex.EnumerateMatches( + TypeNameRegex, + normalized)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var name = typeMatch.Groups["name"].Value; if (isIgnoredName(name)) continue; @@ -266,8 +271,13 @@ public static void EmitDecoratorReferences( var mayBeDecoratorCall = MayBePythonDecoratorCall(preparedLine); if (mayBeDecoratorCall) { - foreach (Match match in DecoratorCallRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + DecoratorCallRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var name = match.Groups["name"].Value; if (isIgnoredName(name)) continue; @@ -290,8 +300,13 @@ public static void EmitDecoratorReferences( if (!mayBeDecoratorCall) { - foreach (Match match in DecoratorRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + DecoratorRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var name = match.Groups["name"].Value; if (isIgnoredName(name)) continue; @@ -319,8 +334,14 @@ private static void EmitDecoratorArgumentReferences( if (argumentStart < 0) return; - foreach (Match identifierMatch in PythonIdentifierRegex.Matches(preparedLine, argumentStart + 1)) + foreach (Match identifierMatch in Regex.EnumerateMatches( + PythonIdentifierRegex, + preparedLine, + argumentStart + 1)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var nameGroup = identifierMatch.Groups["name"]; var name = nameGroup.Value; if (name == decoratorName || isIgnoredName(name) || IsPythonLiteralName(name)) diff --git a/tests/CodeIndex.Tests/BoundedRegexTests.cs b/tests/CodeIndex.Tests/BoundedRegexTests.cs index ea2786e31a..139965dfc4 100644 --- a/tests/CodeIndex.Tests/BoundedRegexTests.cs +++ b/tests/CodeIndex.Tests/BoundedRegexTests.cs @@ -68,6 +68,20 @@ public void EnumerateMatches_StaticPattern_StreamsInMatchOrder() Assert.Equal(["alpha", "beta"], matches); } + [Fact] + public void EnumerateMatches_InstanceRegex_StartsAtRequestedOffset() + { + var regex = new BoundedRegex(@"\w+"); + + var matches = BoundedRegex + .EnumerateMatches(regex, "skip alpha beta", startAt: 5) + .Take(2) + .Select(match => match.Value) + .ToArray(); + + Assert.Equal(["alpha", "beta"], matches); + } + [Fact] public void InstanceMatch_Timeout_ReturnsEmpty() { From a96f13ceead6deb0bf974ed51b68fe2ebda59416 Mon Sep 17 00:00:00 2001 From: Widthdom Date: Sun, 26 Jul 2026 17:56:19 +0900 Subject: [PATCH 16/30] Stream dynamic language reference matches --- DEVELOPER_GUIDE.md | 7 ++ TESTING_GUIDE.md | 2 + ...+stream-dynamic-reference-matches.fixed.md | 26 +++++++ .../Languages/PerlReferenceExtractor.cs | 12 +++- .../PhpReferenceExtractor.LanguageTypes.cs | 68 +++++++++++++++---- .../PhpReferenceExtractor.Members.cs | 7 +- .../Languages/PhpReferenceExtractor.cs | 14 +++- .../RReferenceExtractor.CallsAndResources.cs | 36 ++++++++-- .../Languages/RReferenceExtractor.Members.cs | 25 +++++-- .../Languages/RReferenceExtractor.cs | 21 +++++- .../Languages/RubyReferenceExtractor.cs | 41 +++++++++-- 11 files changed, 222 insertions(+), 37 deletions(-) create mode 100644 changelog.d/unreleased/+stream-dynamic-reference-matches.fixed.md diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index c1501a5d9a..a7dc60927c 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -542,6 +542,10 @@ Python reference scanners stream decorators, annotations, runtime type checks, typing factories, dataclass/attrs integrations, and dynamic imports. The start-offset overload of `BoundedRegex.EnumerateMatches` keeps decorator argument scans demand-driven without rescanning the decorator prefix. +Dynamic-language scanners for PHP, Ruby, R, and Perl stream multi-match +attributes/types, DSL targets, namespace/member/resource references, and arrow +calls. Any loop that writes directly to a bounded reference list must exit at +the cap instead of walking the rest of a dense line. All line-based symbol and reference extractors share `SourceLineSplitter`. It counts newline boundaries once, allocates the exact result array, and then materializes only the line strings that downstream scanners require; do not @@ -3574,6 +3578,9 @@ Python reference scanner は decorator、annotation、runtime type check、typin dataclass / attrs integration、dynamic import を逐次走査する。`BoundedRegex.EnumerateMatches` の start-offset overload により、decorator prefix を再走査せず argument scan も demand-driven のままにする。 +PHP、Ruby、R、Perl の dynamic-language scanner は multi-match の attribute / type、 +DSL target、namespace / member / resource reference、arrow call を逐次走査する。 +bounded reference list へ直接書く loop は dense line の残りを走査せず上限で停止する。 line-based symbol / reference extractor はすべて `SourceLineSplitter` を共有する。 newline boundary を一度数えて exact result array を確保し、downstream scanner が必要とする line string だけを実体化する。`string.Split` による separator-index array を戻してはならない。 diff --git a/TESTING_GUIDE.md b/TESTING_GUIDE.md index 6ca9a9cb99..80b1175158 100644 --- a/TESTING_GUIDE.md +++ b/TESTING_GUIDE.md @@ -591,6 +591,7 @@ Use `docs/test-doc-maintenance-plan.md` before moving oversized suites or adding Functional-language graph fixtures also cover the demand-driven Clojure, Elixir, Erlang, OCAML, and Raku regex loops; keep their bounded-list exit checks adjacent to enumeration. JVM graph fixtures cover demand-driven Java type/module/method-reference, Kotlin type/infix/constructor, Scala contextual, and Gradle/Groovy DSL matches. Preserve dense-line ordering when adding cap-aware exits. Python graph fixtures cover streamed decorator arguments, annotations, runtime type checks, typing factories, dataclass/framework integrations, and dynamic imports. `BoundedRegexTests.EnumerateMatches_InstanceRegex_StartsAtRequestedOffset` fixes the no-prefix-rescan contract used by decorator arguments. + PHP, Ruby, R, and Perl graph fixtures cover streamed attributes/docblocks/types, DSL command targets, namespace/member/resource references, and arrow calls. Keep nested token/type enumeration cap-aware. `ReferenceExtraction_CSharpNoAliasDenseReferences_StaysWithinAllocationBudget` keeps 12,000 no-alias calls from paying for alias dedupe, `CSharpAliasCompaction_DenseDuplicates_StaysWithinAllocationBudget` guards pre-sized stable compaction after alias rewrites, and `MutualRecursion_DenseRepeatedQualifiedNames_StaysWithinAllocationBudget` covers repeated qualified C#- and Python-style cycle names without per-edge normalization strings. `ReusableStatSnapshot_OnePassMaterialization_StaysWithinAllocationBudget` warms a 1,024-row typed snapshot with its exact capacity, then prevents a second candidate collection or redundant path values from returning to the default `net8.0` path. `ReferenceBatchTransactions_RepositoryScaleAtomicFileScopeEliminatesControlledSqlScopes` models 321,352 references distributed over 856 files in the repository snapshot's five/six-batch shape and fixes the control-SQL contract at 5,009 public batch transaction scopes versus zero explicit atomic-file batch scopes; keep it deterministic and allocation-light instead of inserting all modeled rows or asserting elapsed time. For reference-line window performance audits, measure identical prebuilt reference rows against one-batch and 32-batch limits in alternating order, report both elapsed time and `GC.GetAllocatedBytesForCurrentThread`, and remove the timing harness after recording the result; end-to-end `--memory-trace` rebuilds remain corroborating evidence because extraction, graph finalization, and OS page-cache variance can dominate the persistence delta. @@ -1455,6 +1456,7 @@ dotnet test --filter "FullyQualifiedName~GitHelperTests" functional-language graph fixture は Clojure、Elixir、Erlang、OCAML、Raku の demand-driven regex loop も検証します。bounded-list の終了判定は enumeration の直近に維持してください。 JVM graph fixture は Java type / module / method-reference、Kotlin type / infix / constructor、Scala contextual、Gradle / Groovy DSL の demand-driven match を検証します。cap-aware exit を追加するときも dense-line の順序を維持してください。 Python graph fixture は decorator argument、annotation、runtime type check、typing factory、dataclass / framework integration、dynamic import の逐次走査を検証します。`BoundedRegexTests.EnumerateMatches_InstanceRegex_StartsAtRequestedOffset` は decorator argument が prefix を再走査しない契約を固定します。 + PHP、Ruby、R、Perl の graph fixture は attribute / docblock / type、DSL command target、namespace / member / resource reference、arrow call の逐次走査を検証します。nested token / type enumeration も cap-aware のままにしてください。 `ReferenceExtraction_CSharpNoAliasDenseReferences_StaysWithinAllocationBudget` は alias のない 12,000 call が alias dedupe のコストを負わないこと、`CSharpAliasCompaction_DenseDuplicates_StaysWithinAllocationBudget` は alias rewrite 後の事前 capacity 付き stable compaction、`MutualRecursion_DenseRepeatedQualifiedNames_StaysWithinAllocationBudget` は C# / Python 形式の qualified cycle name が edge ごとの正規化文字列を作らないことを保証します。 `ReusableStatSnapshot_OnePassMaterialization_StaysWithinAllocationBudget` は exact capacity を渡した1,024行の typed snapshot を warm-up し、2つ目の候補 collection や重複 path value が通常の `net8.0` 経路へ戻らないようにします。 `ReferenceBatchTransactions_RepositoryScaleAtomicFileScopeEliminatesControlledSqlScopes` は321,352 refsを自己snapshotの5/6 batch形状で856 filesへ分配し、制御SQL契約をpublic batch transaction scope 5,009回対explicit atomic-file scope 0回に固定します。全model rowを挿入したり経過時間をassertしたりせず、deterministicでallocation-lightなまま維持してください。reference-line windowの性能監査では、同一の事前構築済みreference rowを1-batch上限と32-batch上限で交互に測り、経過時間と`GC.GetAllocatedBytesForCurrentThread`の両方を報告して、結果記録後にtiming harnessを削除します。end-to-endの`--memory-trace` rebuildは、extraction・graph finalize・OS page cacheの変動が永続化差を支配し得るため、補助証拠として扱ってください。 diff --git a/changelog.d/unreleased/+stream-dynamic-reference-matches.fixed.md b/changelog.d/unreleased/+stream-dynamic-reference-matches.fixed.md new file mode 100644 index 0000000000..d66612c5ce --- /dev/null +++ b/changelog.d/unreleased/+stream-dynamic-reference-matches.fixed.md @@ -0,0 +1,26 @@ +--- +category: fixed +affected: + - src/CodeIndex/Indexer/References/Languages/PhpReferenceExtractor.cs + - src/CodeIndex/Indexer/References/Languages/PhpReferenceExtractor.LanguageTypes.cs + - src/CodeIndex/Indexer/References/Languages/PhpReferenceExtractor.Members.cs + - src/CodeIndex/Indexer/References/Languages/RubyReferenceExtractor.cs + - src/CodeIndex/Indexer/References/Languages/RReferenceExtractor.cs + - src/CodeIndex/Indexer/References/Languages/RReferenceExtractor.Members.cs + - src/CodeIndex/Indexer/References/Languages/RReferenceExtractor.CallsAndResources.cs + - src/CodeIndex/Indexer/References/Languages/PerlReferenceExtractor.cs + - DEVELOPER_GUIDE.md + - TESTING_GUIDE.md +--- + +## English + +- **Streamed dynamic-language reference matches** — PHP, Ruby, R, and Perl + scanners now enumerate dense attribute, type, DSL, namespace/member/resource, + and arrow-call matches only while the reference budget has capacity. + +## 日本語 + +- **dynamic-language の reference match を逐次走査にしました** — PHP、Ruby、R、 + Perl scanner は dense な attribute、type、DSL、namespace / member / resource、 + arrow-call match を reference budget に空きがある間だけ列挙します。 diff --git a/src/CodeIndex/Indexer/References/Languages/PerlReferenceExtractor.cs b/src/CodeIndex/Indexer/References/Languages/PerlReferenceExtractor.cs index fa178071c2..20e57d3aa5 100644 --- a/src/CodeIndex/Indexer/References/Languages/PerlReferenceExtractor.cs +++ b/src/CodeIndex/Indexer/References/Languages/PerlReferenceExtractor.cs @@ -231,8 +231,11 @@ private static void EmitArrowCallReferences( Func resolveContainerForCall, Action addCallLikeReference) { - foreach (Match match in ArrowCallRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches(ArrowCallRegex, preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var nameGroup = match.Groups["name"]; addCallLikeReference(nameGroup.Value, nameGroup.Index); @@ -262,8 +265,13 @@ private static void EmitQualifiedFunctionCallReferences( int lineNumber, Func resolveContainerForCall) { - foreach (Match match in QualifiedFunctionCallRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + QualifiedFunctionCallRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var nameGroup = match.Groups["name"]; if (IsQualifiedSubroutineDefinition(preparedLine, nameGroup.Index)) continue; diff --git a/src/CodeIndex/Indexer/References/Languages/PhpReferenceExtractor.LanguageTypes.cs b/src/CodeIndex/Indexer/References/Languages/PhpReferenceExtractor.LanguageTypes.cs index a3eaed9aba..f577c35070 100644 --- a/src/CodeIndex/Indexer/References/Languages/PhpReferenceExtractor.LanguageTypes.cs +++ b/src/CodeIndex/Indexer/References/Languages/PhpReferenceExtractor.LanguageTypes.cs @@ -18,8 +18,11 @@ public static void EmitAttributeReferences( if (!preparedLine.Contains("#[", StringComparison.Ordinal)) return; - foreach (Match match in AttributeRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches(AttributeRegex, preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var nameGroup = match.Groups["name"]; var rawName = nameGroup.Value; var leadingBackslashCount = 0; @@ -74,8 +77,11 @@ public static void EmitStaticAccessReferences( if (preparedLine.IndexOf("::", StringComparison.Ordinal) < 0) return; - foreach (Match match in StaticAccessRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches(StaticAccessRegex, preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var nameGroup = match.Groups["name"]; var rawName = nameGroup.Value; var leadingBackslashCount = 0; @@ -153,8 +159,11 @@ public static void EmitInstanceofReferences( if (preparedLine.IndexOf("instanceof", StringComparison.OrdinalIgnoreCase) < 0) return; - foreach (Match match in InstanceofRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches(InstanceofRegex, preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + AddPhpTypeReferenceFromQualifiedName( match.Groups["name"], references, @@ -178,8 +187,11 @@ public static void EmitCatchTypeReferences( if (preparedLine.IndexOf("catch", StringComparison.OrdinalIgnoreCase) < 0) return; - foreach (Match match in CatchTypeRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches(CatchTypeRegex, preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + foreach (Capture capture in match.Groups["name"].Captures) { AddPhpTypeReferenceFromQualifiedName( @@ -209,8 +221,11 @@ public static void EmitReturnTypeReferences( return; } - foreach (Match match in ReturnTypeRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches(ReturnTypeRegex, preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + foreach (Capture capture in match.Groups["name"].Captures) { if (IsPhpBuiltinTypeName(capture.Value)) @@ -240,8 +255,11 @@ public static void EmitParameterTypeReferences( if (preparedLine.IndexOf('$') < 0) return; - foreach (Match match in ParameterTypeRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches(ParameterTypeRegex, preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + foreach (Capture capture in match.Groups["name"].Captures) { if (IsPhpBuiltinTypeName(capture.Value)) @@ -277,8 +295,11 @@ public static void EmitPropertyTypeReferences( return; } - foreach (Match match in PropertyTypeRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches(PropertyTypeRegex, preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + foreach (Capture capture in match.Groups["name"].Captures) { if (IsPhpBuiltinTypeName(capture.Value)) @@ -311,8 +332,11 @@ public static void EmitInheritanceTypeReferences( return; } - foreach (Match match in InheritanceTypeRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches(InheritanceTypeRegex, preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + foreach (Capture capture in match.Groups["name"].Captures) { AddPhpTypeReferenceFromQualifiedName( @@ -397,8 +421,13 @@ public static void EmitUseFunctionReferences( return; var importsGroup = match.Groups["imports"]; - foreach (Match itemMatch in UseImportItemRegex.Matches(importsGroup.Value)) + foreach (Match itemMatch in Regex.EnumerateMatches( + UseImportItemRegex, + importsGroup.Value)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var itemGroup = itemMatch.Groups["name"]; AddPhpReferenceFromName( itemGroup.Value, @@ -447,8 +476,13 @@ public static void EmitUseConstReferences( return; var importsGroup = match.Groups["imports"]; - foreach (Match itemMatch in UseImportItemRegex.Matches(importsGroup.Value)) + foreach (Match itemMatch in Regex.EnumerateMatches( + UseImportItemRegex, + importsGroup.Value)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var itemGroup = itemMatch.Groups["name"]; AddPhpReferenceFromName( itemGroup.Value, @@ -482,8 +516,13 @@ private static void EmitGroupUseTypeReferences( var prefix = prefixEnd == rawPrefix.Length ? rawPrefix : rawPrefix.Substring(0, prefixEnd); var itemsGroup = groupMatch.Groups["items"]; - foreach (Match itemMatch in GroupUseTypeItemRegex.Matches(itemsGroup.Value)) + foreach (Match itemMatch in Regex.EnumerateMatches( + GroupUseTypeItemRegex, + itemsGroup.Value)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + if (itemMatch.Groups["kind"].Success) continue; @@ -532,8 +571,13 @@ private static void EmitGroupUseImportReferences( var prefix = prefixEnd == rawPrefix.Length ? rawPrefix : rawPrefix.Substring(0, prefixEnd); var itemsGroup = groupMatch.Groups["items"]; - foreach (Match itemMatch in GroupUseTypeItemRegex.Matches(itemsGroup.Value)) + foreach (Match itemMatch in Regex.EnumerateMatches( + GroupUseTypeItemRegex, + itemsGroup.Value)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var isTargetKind = itemMatch.Groups["kind"].Success && itemMatch.Groups["kind"].Value.Equals(importKind, StringComparison.OrdinalIgnoreCase); if (requireImportKind != isTargetKind) diff --git a/src/CodeIndex/Indexer/References/Languages/PhpReferenceExtractor.Members.cs b/src/CodeIndex/Indexer/References/Languages/PhpReferenceExtractor.Members.cs index 5124ba3964..f72538b8e2 100644 --- a/src/CodeIndex/Indexer/References/Languages/PhpReferenceExtractor.Members.cs +++ b/src/CodeIndex/Indexer/References/Languages/PhpReferenceExtractor.Members.cs @@ -123,8 +123,13 @@ public static void EmitObjectMemberAccessReferences( return; } - foreach (Match match in ObjectMemberAccessRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + ObjectMemberAccessRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var nameGroup = match.Groups["name"]; ReferenceExtractor.AddReference( references, diff --git a/src/CodeIndex/Indexer/References/Languages/PhpReferenceExtractor.cs b/src/CodeIndex/Indexer/References/Languages/PhpReferenceExtractor.cs index e3c3301874..554bb266d5 100644 --- a/src/CodeIndex/Indexer/References/Languages/PhpReferenceExtractor.cs +++ b/src/CodeIndex/Indexer/References/Languages/PhpReferenceExtractor.cs @@ -348,8 +348,13 @@ public static void EmitDocblockMethodParameterTypeReferences( return; var paramsGroup = match.Groups["params"]; - foreach (Match parameterMatch in DocblockMethodParameterTypeRegex.Matches(paramsGroup.Value)) + foreach (Match parameterMatch in Regex.EnumerateMatches( + DocblockMethodParameterTypeRegex, + paramsGroup.Value)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var typesGroup = parameterMatch.Groups["types"]; EmitDocblockTypeGroupReferences( typesGroup.Value, @@ -456,8 +461,13 @@ private static void EmitDocblockTypeGroupReferences( int lineNumber, SymbolRecord? container) { - foreach (Match typeMatch in DocblockTypeNameRegex.Matches(typeExpression)) + foreach (Match typeMatch in Regex.EnumerateMatches( + DocblockTypeNameRegex, + typeExpression)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var nameGroup = typeMatch.Groups["name"]; if (IsPhpBuiltinTypeName(nameGroup.Value)) continue; diff --git a/src/CodeIndex/Indexer/References/Languages/RReferenceExtractor.CallsAndResources.cs b/src/CodeIndex/Indexer/References/Languages/RReferenceExtractor.CallsAndResources.cs index b50d451280..66569359aa 100644 --- a/src/CodeIndex/Indexer/References/Languages/RReferenceExtractor.CallsAndResources.cs +++ b/src/CodeIndex/Indexer/References/Languages/RReferenceExtractor.CallsAndResources.cs @@ -19,8 +19,11 @@ public static void EmitBacktickCallReferences( if (preparedLine.IndexOf('`') < 0 || preparedLine.IndexOf('(') < 0) return; - foreach (Match match in BacktickCallRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches(BacktickCallRegex, preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var nameGroup = match.Groups["name"]; var name = nameGroup.Value; if (definitionNames != null && definitionNames.Contains(name)) @@ -52,8 +55,13 @@ public static void EmitInfixOperatorCallReferences( if (preparedLine.IndexOf('%') < 0) return; - foreach (Match match in InfixOperatorCallRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + InfixOperatorCallRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var nameGroup = match.Groups["name"]; var name = nameGroup.Value; if (definitionNames != null && definitionNames.Contains(name)) @@ -171,8 +179,11 @@ public static void EmitDataCallReferences( return; var line = StripRNamespaceDirectiveComment(originalLine); - foreach (Match match in DataCallDatasetRegex.Matches(line)) + foreach (Match match in Regex.EnumerateMatches(DataCallDatasetRegex, line)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var name = match.Groups["name"]; ReferenceExtractor.AddReference( references, @@ -226,8 +237,13 @@ public static void EmitSystemFileReferences( return; var line = StripRNamespaceDirectiveComment(originalLine); - foreach (Match match in SystemFilePathPartRegex.Matches(line)) + foreach (Match match in Regex.EnumerateMatches( + SystemFilePathPartRegex, + line)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var name = match.Groups["name"]; ReferenceExtractor.AddReference( references, @@ -333,8 +349,11 @@ private static void EmitDocumentationTopicReferences( return; var line = StripRNamespaceDirectiveComment(originalLine); - foreach (Match match in DocumentationTopicRegex.Matches(line)) + foreach (Match match in Regex.EnumerateMatches(DocumentationTopicRegex, line)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var name = match.Groups["name"]; ReferenceExtractor.AddReference( references, @@ -467,8 +486,13 @@ private static void EmitPackageNameArgumentReferences( return; var line = StripRNamespaceDirectiveComment(originalLine); - foreach (Match match in InstallPackagesNameRegex.Matches(line)) + foreach (Match match in Regex.EnumerateMatches( + InstallPackagesNameRegex, + line)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var name = match.Groups["name"]; ReferenceExtractor.AddReference( references, diff --git a/src/CodeIndex/Indexer/References/Languages/RReferenceExtractor.Members.cs b/src/CodeIndex/Indexer/References/Languages/RReferenceExtractor.Members.cs index 1eb7c0415c..59b5b14aa8 100644 --- a/src/CodeIndex/Indexer/References/Languages/RReferenceExtractor.Members.cs +++ b/src/CodeIndex/Indexer/References/Languages/RReferenceExtractor.Members.cs @@ -19,8 +19,13 @@ public static void EmitDollarMemberReferences( if (preparedLine.IndexOf('$') < 0) return; - foreach (Match match in DollarMemberReferenceRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + DollarMemberReferenceRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var backtickReceiverGroup = match.Groups["backtickReceiver"]; var receiverGroup = backtickReceiverGroup.Success ? backtickReceiverGroup : match.Groups["receiver"]; var receiver = receiverGroup.Value; @@ -73,8 +78,13 @@ public static void EmitBracketMemberReferences( return; var line = StripRNamespaceDirectiveComment(originalLine); - foreach (Match match in BracketMemberReferenceRegex.Matches(line)) + foreach (Match match in Regex.EnumerateMatches( + BracketMemberReferenceRegex, + line)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var backtickReceiverGroup = match.Groups["backtickReceiver"]; var receiverGroup = backtickReceiverGroup.Success ? backtickReceiverGroup : match.Groups["receiver"]; var receiver = receiverGroup.Value; @@ -121,8 +131,13 @@ public static void EmitSlotMemberReferences( if (preparedLine.IndexOf('@') < 0) return; - foreach (Match match in SlotMemberReferenceRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + SlotMemberReferenceRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var backtickReceiverGroup = match.Groups["backtickReceiver"]; var receiverGroup = backtickReceiverGroup.Success ? backtickReceiverGroup : match.Groups["receiver"]; var receiver = receiverGroup.Value; @@ -159,7 +174,9 @@ public static void EmitSlotMemberReferences( private static IEnumerable<(string Name, int Index)> EnumerateNamespaceDirectiveNames(string value, int baseIndex) { - foreach (Match match in NamespaceDirectiveNameRegex.Matches(value)) + foreach (Match match in Regex.EnumerateMatches( + NamespaceDirectiveNameRegex, + value)) { var backtickNameGroup = match.Groups["backtickName"]; var nameGroup = backtickNameGroup.Success ? backtickNameGroup : match.Groups["name"]; diff --git a/src/CodeIndex/Indexer/References/Languages/RReferenceExtractor.cs b/src/CodeIndex/Indexer/References/Languages/RReferenceExtractor.cs index 2967bd3f6c..25f9c36691 100644 --- a/src/CodeIndex/Indexer/References/Languages/RReferenceExtractor.cs +++ b/src/CodeIndex/Indexer/References/Languages/RReferenceExtractor.cs @@ -134,8 +134,13 @@ public static void EmitNamespaceReferences( if (preparedLine.IndexOf("::", StringComparison.Ordinal) < 0) return; - foreach (Match match in NamespaceReferenceRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + NamespaceReferenceRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var package = match.Groups["package"].Value; var separator = match.Groups["sep"].Value; var backtickNameGroup = match.Groups["backtickName"]; @@ -327,8 +332,13 @@ public static void EmitNamespaceDirectiveReferences( var routinesStart = useDynLibMatch.Index + useDynLibMatch.Length; var routines = directiveLine[routinesStart..]; - foreach (Match routineMatch in NamespaceUseDynLibRoutineRegex.Matches(routines)) + foreach (Match routineMatch in Regex.EnumerateMatches( + NamespaceUseDynLibRoutineRegex, + routines)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var routine = GetNamespaceDirectiveToken( routineMatch, "backtickName", @@ -530,8 +540,13 @@ public static void EmitS4DispatchReferences( return; var signatureBody = signatureCall.Groups["body"]; - foreach (Match signatureMatch in S4SignatureClassRegex.Matches(signatureBody.Value)) + foreach (Match signatureMatch in Regex.EnumerateMatches( + S4SignatureClassRegex, + signatureBody.Value)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var classToken = GetNamespaceDirectiveToken( signatureMatch, "backtickName", diff --git a/src/CodeIndex/Indexer/References/Languages/RubyReferenceExtractor.cs b/src/CodeIndex/Indexer/References/Languages/RubyReferenceExtractor.cs index 8ad41402a5..445281f09b 100644 --- a/src/CodeIndex/Indexer/References/Languages/RubyReferenceExtractor.cs +++ b/src/CodeIndex/Indexer/References/Languages/RubyReferenceExtractor.cs @@ -92,8 +92,11 @@ public static void EmitAdditionalCallReferences( lineNumber, resolveContainerForCall); - foreach (Match match in CommandCallRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches(CommandCallRegex, preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var name = match.Groups["name"].Value; var callIndex = match.Groups["name"].Index; matchedCallIndices.Add(callIndex); @@ -119,8 +122,11 @@ public static void EmitAdditionalCallReferences( lineNumber, resolveContainerForCall); - foreach (Match match in BlockCallRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches(BlockCallRegex, preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var name = match.Groups["name"].Value; var callIndex = match.Groups["name"].Index; addCallLikeReference(name, callIndex); @@ -173,8 +179,11 @@ public static void EmitCommandTargetReferences( resolveContainerForCall); var matchedAny = false; - foreach (Match match in CommandTargetTokenRegex.Matches(tail)) + foreach (Match match in Regex.EnumerateMatches(CommandTargetTokenRegex, tail)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var rawToken = match.Groups["token"].Value; if (rawToken.Length == 0) continue; @@ -250,8 +259,13 @@ private static void EmitSymbolLiteralFirstArgumentReferences( int lineNumber, Func resolveContainerForCall) { - foreach (Match match in SymbolLiteralFirstArgumentRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + SymbolLiteralFirstArgumentRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var rawToken = match.Groups["token"].Value; var token = NormalizeCommandTargetToken(rawToken); if (string.IsNullOrWhiteSpace(token)) @@ -310,11 +324,21 @@ private static void EmitRescueTypeReferences( int lineNumber, Func resolveContainerForCall) { - foreach (Match rescueMatch in RescueClauseRegex.Matches(preparedLine)) + foreach (Match rescueMatch in Regex.EnumerateMatches( + RescueClauseRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var typesGroup = rescueMatch.Groups["types"]; - foreach (Match typeMatch in QualifiedConstantRegex.Matches(typesGroup.Value)) + foreach (Match typeMatch in Regex.EnumerateMatches( + QualifiedConstantRegex, + typesGroup.Value)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var name = typeMatch.Value; var tokenIndex = typesGroup.Index + typeMatch.Index; var targetContainer = resolveContainerForCall(tokenIndex); @@ -346,8 +370,11 @@ private static void EmitClassNameOptionReferences( if (!ClassNameOptionCommandNames.Contains(commandName)) return; - foreach (Match match in ClassNameOptionRegex.Matches(tail)) + foreach (Match match in Regex.EnumerateMatches(ClassNameOptionRegex, tail)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var name = match.Groups["name"].Value; var tokenIndex = tailStartIndex + match.Groups["name"].Index; var targetContainer = resolveContainerForCall(tokenIndex); From 0ae9365734e7fc928d0ce93c0add1a43adb84088 Mon Sep 17 00:00:00 2001 From: Widthdom Date: Sun, 26 Jul 2026 18:01:17 +0900 Subject: [PATCH 17/30] Stream systems language reference matches --- DEVELOPER_GUIDE.md | 8 +++ TESTING_GUIDE.md | 2 + ...+stream-systems-reference-matches.fixed.md | 29 +++++++++ .../Languages/CppReferenceExtractor.cs | 16 ++++- .../Languages/GoReferenceExtractor.cs | 19 +++++- .../RustReferenceExtractor.SignatureTypes.cs | 7 +- .../RustReferenceExtractor.ValueTypes.cs | 21 +++++- .../Languages/RustReferenceExtractor.cs | 27 ++++++-- .../ScientificNativeReferenceEmitter.cs | 8 ++- .../Languages/SwiftReferenceExtractor.cs | 7 +- ...eferenceExtractionSupport.CppTypeGroups.cs | 65 +++++++++++++++---- .../LanguageReferenceExtractionSupport.Go.cs | 7 +- ...renceExtractionSupport.GoCompositeTypes.cs | 19 +++++- ...ReferenceExtractionSupport.GoSignatures.cs | 4 +- 14 files changed, 204 insertions(+), 35 deletions(-) create mode 100644 changelog.d/unreleased/+stream-systems-reference-matches.fixed.md diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index a7dc60927c..4bc11c0802 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -546,6 +546,10 @@ Dynamic-language scanners for PHP, Ruby, R, and Perl stream multi-match attributes/types, DSL targets, namespace/member/resource references, and arrow calls. Any loop that writes directly to a bounded reference list must exit at the cap instead of walking the rest of a dense line. +Systems-language scanners stream C/C++ construction and template groups, Rust +calls and value/signature types, Swift property wrappers, Go concurrency and +composite/signature types, plus shared scientific/native call groups. Preserve +source-order emission and stop owned bounded lists at capacity. All line-based symbol and reference extractors share `SourceLineSplitter`. It counts newline boundaries once, allocates the exact result array, and then materializes only the line strings that downstream scanners require; do not @@ -3581,6 +3585,10 @@ demand-driven のままにする。 PHP、Ruby、R、Perl の dynamic-language scanner は multi-match の attribute / type、 DSL target、namespace / member / resource reference、arrow call を逐次走査する。 bounded reference list へ直接書く loop は dense line の残りを走査せず上限で停止する。 +systems-language scanner は C / C++ construction と template group、Rust call と +value / signature type、Swift property wrapper、Go concurrency と composite / signature +type、共有 scientific / native call group を逐次走査する。source-order emission を維持し、 +所有する bounded list は上限で停止する。 line-based symbol / reference extractor はすべて `SourceLineSplitter` を共有する。 newline boundary を一度数えて exact result array を確保し、downstream scanner が必要とする line string だけを実体化する。`string.Split` による separator-index array を戻してはならない。 diff --git a/TESTING_GUIDE.md b/TESTING_GUIDE.md index 80b1175158..60177baa37 100644 --- a/TESTING_GUIDE.md +++ b/TESTING_GUIDE.md @@ -592,6 +592,7 @@ Use `docs/test-doc-maintenance-plan.md` before moving oversized suites or adding JVM graph fixtures cover demand-driven Java type/module/method-reference, Kotlin type/infix/constructor, Scala contextual, and Gradle/Groovy DSL matches. Preserve dense-line ordering when adding cap-aware exits. Python graph fixtures cover streamed decorator arguments, annotations, runtime type checks, typing factories, dataclass/framework integrations, and dynamic imports. `BoundedRegexTests.EnumerateMatches_InstanceRegex_StartsAtRequestedOffset` fixes the no-prefix-rescan contract used by decorator arguments. PHP, Ruby, R, and Perl graph fixtures cover streamed attributes/docblocks/types, DSL command targets, namespace/member/resource references, and arrow calls. Keep nested token/type enumeration cap-aware. + Systems-language graph fixtures cover streamed C/C++ friend/construction/template groups, Rust macro/value/signature types, Swift wrappers, Go concurrency/composite/signature types, and shared scientific/native call groups. `ReferenceExtraction_CSharpNoAliasDenseReferences_StaysWithinAllocationBudget` keeps 12,000 no-alias calls from paying for alias dedupe, `CSharpAliasCompaction_DenseDuplicates_StaysWithinAllocationBudget` guards pre-sized stable compaction after alias rewrites, and `MutualRecursion_DenseRepeatedQualifiedNames_StaysWithinAllocationBudget` covers repeated qualified C#- and Python-style cycle names without per-edge normalization strings. `ReusableStatSnapshot_OnePassMaterialization_StaysWithinAllocationBudget` warms a 1,024-row typed snapshot with its exact capacity, then prevents a second candidate collection or redundant path values from returning to the default `net8.0` path. `ReferenceBatchTransactions_RepositoryScaleAtomicFileScopeEliminatesControlledSqlScopes` models 321,352 references distributed over 856 files in the repository snapshot's five/six-batch shape and fixes the control-SQL contract at 5,009 public batch transaction scopes versus zero explicit atomic-file batch scopes; keep it deterministic and allocation-light instead of inserting all modeled rows or asserting elapsed time. For reference-line window performance audits, measure identical prebuilt reference rows against one-batch and 32-batch limits in alternating order, report both elapsed time and `GC.GetAllocatedBytesForCurrentThread`, and remove the timing harness after recording the result; end-to-end `--memory-trace` rebuilds remain corroborating evidence because extraction, graph finalization, and OS page-cache variance can dominate the persistence delta. @@ -1457,6 +1458,7 @@ dotnet test --filter "FullyQualifiedName~GitHelperTests" JVM graph fixture は Java type / module / method-reference、Kotlin type / infix / constructor、Scala contextual、Gradle / Groovy DSL の demand-driven match を検証します。cap-aware exit を追加するときも dense-line の順序を維持してください。 Python graph fixture は decorator argument、annotation、runtime type check、typing factory、dataclass / framework integration、dynamic import の逐次走査を検証します。`BoundedRegexTests.EnumerateMatches_InstanceRegex_StartsAtRequestedOffset` は decorator argument が prefix を再走査しない契約を固定します。 PHP、Ruby、R、Perl の graph fixture は attribute / docblock / type、DSL command target、namespace / member / resource reference、arrow call の逐次走査を検証します。nested token / type enumeration も cap-aware のままにしてください。 + systems-language graph fixture は C / C++ friend / construction / template group、Rust macro / value / signature type、Swift wrapper、Go concurrency / composite / signature type、共有 scientific / native call group の逐次走査を検証します。 `ReferenceExtraction_CSharpNoAliasDenseReferences_StaysWithinAllocationBudget` は alias のない 12,000 call が alias dedupe のコストを負わないこと、`CSharpAliasCompaction_DenseDuplicates_StaysWithinAllocationBudget` は alias rewrite 後の事前 capacity 付き stable compaction、`MutualRecursion_DenseRepeatedQualifiedNames_StaysWithinAllocationBudget` は C# / Python 形式の qualified cycle name が edge ごとの正規化文字列を作らないことを保証します。 `ReusableStatSnapshot_OnePassMaterialization_StaysWithinAllocationBudget` は exact capacity を渡した1,024行の typed snapshot を warm-up し、2つ目の候補 collection や重複 path value が通常の `net8.0` 経路へ戻らないようにします。 `ReferenceBatchTransactions_RepositoryScaleAtomicFileScopeEliminatesControlledSqlScopes` は321,352 refsを自己snapshotの5/6 batch形状で856 filesへ分配し、制御SQL契約をpublic batch transaction scope 5,009回対explicit atomic-file scope 0回に固定します。全model rowを挿入したり経過時間をassertしたりせず、deterministicでallocation-lightなまま維持してください。reference-line windowの性能監査では、同一の事前構築済みreference rowを1-batch上限と32-batch上限で交互に測り、経過時間と`GC.GetAllocatedBytesForCurrentThread`の両方を報告して、結果記録後にtiming harnessを削除します。end-to-endの`--memory-trace` rebuildは、extraction・graph finalize・OS page cacheの変動が永続化差を支配し得るため、補助証拠として扱ってください。 diff --git a/changelog.d/unreleased/+stream-systems-reference-matches.fixed.md b/changelog.d/unreleased/+stream-systems-reference-matches.fixed.md new file mode 100644 index 0000000000..40be258a47 --- /dev/null +++ b/changelog.d/unreleased/+stream-systems-reference-matches.fixed.md @@ -0,0 +1,29 @@ +--- +category: fixed +affected: + - src/CodeIndex/Indexer/References/Languages/CppReferenceExtractor.cs + - src/CodeIndex/Indexer/References/Languages/GoReferenceExtractor.cs + - src/CodeIndex/Indexer/References/Languages/RustReferenceExtractor.cs + - src/CodeIndex/Indexer/References/Languages/RustReferenceExtractor.SignatureTypes.cs + - src/CodeIndex/Indexer/References/Languages/RustReferenceExtractor.ValueTypes.cs + - src/CodeIndex/Indexer/References/Languages/SwiftReferenceExtractor.cs + - src/CodeIndex/Indexer/References/Languages/ScientificNativeReferenceEmitter.cs + - src/CodeIndex/Indexer/References/Support/LanguageReferenceExtractionSupport.CppTypeGroups.cs + - src/CodeIndex/Indexer/References/Support/LanguageReferenceExtractionSupport.Go.cs + - src/CodeIndex/Indexer/References/Support/LanguageReferenceExtractionSupport.GoCompositeTypes.cs + - src/CodeIndex/Indexer/References/Support/LanguageReferenceExtractionSupport.GoSignatures.cs + - DEVELOPER_GUIDE.md + - TESTING_GUIDE.md +--- + +## English + +- **Streamed systems-language reference matches** — C/C++, Rust, Swift, Go, + and shared scientific/native scanners now consume dense construction, type, + concurrency, wrapper, and call matches on demand. + +## 日本語 + +- **systems-language の reference match を逐次走査にしました** — C / C++、Rust、 + Swift、Go と共有 scientific / native scanner は dense な construction、type、 + concurrency、wrapper、call match を demand-driven に消費します。 diff --git a/src/CodeIndex/Indexer/References/Languages/CppReferenceExtractor.cs b/src/CodeIndex/Indexer/References/Languages/CppReferenceExtractor.cs index 68c4d340d4..2bb1a08cd1 100644 --- a/src/CodeIndex/Indexer/References/Languages/CppReferenceExtractor.cs +++ b/src/CodeIndex/Indexer/References/Languages/CppReferenceExtractor.cs @@ -53,11 +53,23 @@ private static void EmitFriendReferences( return; } - foreach (Match match in FriendTypeRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches(FriendTypeRegex, preparedLine)) + { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + AddFriendReference(match.Groups["name"]); + } + + foreach (Match match in Regex.EnumerateMatches( + FriendFunctionRegex, + preparedLine)) + { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; - foreach (Match match in FriendFunctionRegex.Matches(preparedLine)) AddFriendReference(match.Groups["name"]); + } void AddFriendReference(Group group) { diff --git a/src/CodeIndex/Indexer/References/Languages/GoReferenceExtractor.cs b/src/CodeIndex/Indexer/References/Languages/GoReferenceExtractor.cs index 8527307af5..5043ac531f 100644 --- a/src/CodeIndex/Indexer/References/Languages/GoReferenceExtractor.cs +++ b/src/CodeIndex/Indexer/References/Languages/GoReferenceExtractor.cs @@ -32,8 +32,13 @@ public static void EmitConcurrencyReferences( { if (preparedLine.IndexOf("go", StringComparison.Ordinal) >= 0) { - foreach (Match match in GoroutineCallRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + GoroutineCallRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var group = match.Groups["name"]; var rawName = group.Value; var dot = rawName.LastIndexOf('.'); @@ -55,8 +60,11 @@ public static void EmitConcurrencyReferences( if (preparedLine.IndexOf("<-", StringComparison.Ordinal) < 0) return; - foreach (Match match in ChannelSendRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches(ChannelSendRegex, preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + ReferenceExtractor.AddReference( references, seen, @@ -69,8 +77,13 @@ public static void EmitConcurrencyReferences( resolveContainerForColumn(match.Groups["name"].Index)); } - foreach (Match match in ChannelReceiveRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + ChannelReceiveRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + if (!IsGoChannelReceiveArrow(preparedLine, match.Index)) continue; diff --git a/src/CodeIndex/Indexer/References/Languages/RustReferenceExtractor.SignatureTypes.cs b/src/CodeIndex/Indexer/References/Languages/RustReferenceExtractor.SignatureTypes.cs index 9f58c16d15..e6676ac3a3 100644 --- a/src/CodeIndex/Indexer/References/Languages/RustReferenceExtractor.SignatureTypes.cs +++ b/src/CodeIndex/Indexer/References/Languages/RustReferenceExtractor.SignatureTypes.cs @@ -58,8 +58,13 @@ private static void EmitMutableReferenceTypeReferences( return; } - foreach (Match match in MutableReferenceTypeRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + MutableReferenceTypeRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + if (!IsMutableReferenceTypeContext(preparedLine, match.Index)) continue; diff --git a/src/CodeIndex/Indexer/References/Languages/RustReferenceExtractor.ValueTypes.cs b/src/CodeIndex/Indexer/References/Languages/RustReferenceExtractor.ValueTypes.cs index cf647a9640..6fa0bfbde7 100644 --- a/src/CodeIndex/Indexer/References/Languages/RustReferenceExtractor.ValueTypes.cs +++ b/src/CodeIndex/Indexer/References/Languages/RustReferenceExtractor.ValueTypes.cs @@ -197,8 +197,13 @@ private static void EmitAssociatedCallReceiverTypeReferences( return; } - foreach (Match match in AssociatedCallReceiverRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + AssociatedCallReceiverRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var receiverGroup = match.Groups["receiver"]; var receiver = receiverGroup.Value; var leafStart = receiver.LastIndexOf("::", StringComparison.Ordinal); @@ -250,8 +255,13 @@ private static void EmitAssociatedValueReceiverTypeReferences( return; } - foreach (Match match in AssociatedValueReceiverRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + AssociatedValueReceiverRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var receiverGroup = match.Groups["receiver"]; var receiver = receiverGroup.Value; var leafStart = receiver.LastIndexOf("::", StringComparison.Ordinal); @@ -423,8 +433,13 @@ private static void EmitStructLiteralInstantiationReferences( return; } - foreach (Match match in StructLiteralRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + StructLiteralRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var nameGroup = match.Groups["name"]; var name = nameGroup.Value; var leafStart = name.LastIndexOf("::", StringComparison.Ordinal); diff --git a/src/CodeIndex/Indexer/References/Languages/RustReferenceExtractor.cs b/src/CodeIndex/Indexer/References/Languages/RustReferenceExtractor.cs index 2deb32bdf9..82ace8f5e2 100644 --- a/src/CodeIndex/Indexer/References/Languages/RustReferenceExtractor.cs +++ b/src/CodeIndex/Indexer/References/Languages/RustReferenceExtractor.cs @@ -360,7 +360,9 @@ public static void EmitAdditionalCallReferences(string preparedLine, Action= 0) { - foreach (Match match in RawIdentifierCallRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + RawIdentifierCallRegex, + preparedLine)) { var name = match.Groups["name"].Value; var callIndex = match.Groups["name"].Index; @@ -371,7 +373,7 @@ public static void EmitAdditionalCallReferences(string preparedLine, Action addCallLikeReference) { - foreach (Match match in regex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches(regex, preparedLine)) { var group = match.Groups["name"]; addCallLikeReference(group.Value, group.Index); @@ -171,7 +171,9 @@ private void EmitCallMatches( private void EmitAdaBareCalls() { - foreach (Match bareCall in AdaBareCallRegex.Matches(preparedLine)) + foreach (Match bareCall in Regex.EnumerateMatches( + AdaBareCallRegex, + preparedLine)) { var group = bareCall.Groups["name"]; var separatorIndex = group.Value.LastIndexOf('.'); @@ -222,7 +224,7 @@ private void EmitMatches( string referenceKind, bool normalizeQualifiedTypeName = false) { - foreach (Match match in regex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches(regex, preparedLine)) { EmitGroup( match.Groups["name"], diff --git a/src/CodeIndex/Indexer/References/Languages/SwiftReferenceExtractor.cs b/src/CodeIndex/Indexer/References/Languages/SwiftReferenceExtractor.cs index cf5be4ad6b..8b0c4290e1 100644 --- a/src/CodeIndex/Indexer/References/Languages/SwiftReferenceExtractor.cs +++ b/src/CodeIndex/Indexer/References/Languages/SwiftReferenceExtractor.cs @@ -458,8 +458,13 @@ private static void EmitPropertyWrapperTypeReferences( if (!attributes.Success || attributes.Length == 0) return; - foreach (Match attributeMatch in PropertyWrapperAttributeRegex.Matches(attributes.Value)) + foreach (Match attributeMatch in Regex.EnumerateMatches( + PropertyWrapperAttributeRegex, + attributes.Value)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var nameGroup = attributeMatch.Groups["name"]; if (!nameGroup.Success) continue; diff --git a/src/CodeIndex/Indexer/References/Support/LanguageReferenceExtractionSupport.CppTypeGroups.cs b/src/CodeIndex/Indexer/References/Support/LanguageReferenceExtractionSupport.CppTypeGroups.cs index be746f74ea..a8cf7434df 100644 --- a/src/CodeIndex/Indexer/References/Support/LanguageReferenceExtractionSupport.CppTypeGroups.cs +++ b/src/CodeIndex/Indexer/References/Support/LanguageReferenceExtractionSupport.CppTypeGroups.cs @@ -20,11 +20,18 @@ private sealed class CppTypeReferenceLineContext( internal string Language { get; } = language; internal string PreparedLine { get; } = preparedLine; internal string OriginalLine { get; } = originalLine; + internal bool LimitReached => + ReferenceExtractor.ReferenceLimitReached(references); internal void EmitTypeExpressions(Regex regex) { - foreach (Match match in regex.Matches(PreparedLine)) + foreach (Match match in Regex.EnumerateMatches(regex, PreparedLine)) + { + if (LimitReached) + break; + AddTypeExpression(match.Groups["type"]); + } } internal void AddTypeExpression(Group group) @@ -130,8 +137,15 @@ private static void EmitCppHeaderConstructionAndCastReferences( if (ContainsOrdinalKeyword(preparedLine, "new")) { - foreach (Match match in CppNewTypeRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + CppNewTypeRegex, + preparedLine)) + { + if (line.LimitReached) + break; + line.AddInstantiation(match.Groups["type"]); + } } if (preparedLine.IndexOf("_cast", StringComparison.Ordinal) >= 0) @@ -435,17 +449,28 @@ private static void EmitCppOperandConstructionAndAliasReferences( || preparedLine.IndexOf('=') >= 0); if (hasBraceConstruction) { - foreach (Match match in CppBraceConstructionRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + CppBraceConstructionRegex, + preparedLine)) + { + if (line.LimitReached) + break; + line.AddInstantiation(match.Groups["type"]); + } } var hasScopeSeparator = preparedLine.IndexOf("::", StringComparison.Ordinal) >= 0; if (hasBraceConstruction && hasTemplateOpen && hasScopeSeparator) { - foreach (Match match in - CppQualifiedTemplateBraceConstructionRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + CppQualifiedTemplateBraceConstructionRegex, + preparedLine)) { + if (line.LimitReached) + break; + var group = match.Groups["args"]; line.AddTypeExpression(group); } @@ -469,9 +494,13 @@ private static void EmitCppOperandConstructionAndAliasReferences( && (ContainsOrdinalKeyword(preparedLine, "class") || ContainsOrdinalKeyword(preparedLine, "struct"))) { - foreach (Match match in - CppExplicitTemplateInstantiationRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + CppExplicitTemplateInstantiationRegex, + preparedLine)) { + if (line.LimitReached) + break; + line.AddInstantiation(match.Groups["type"]); } } @@ -497,9 +526,13 @@ private static void EmitCppConstraintAndDeclarationReferences( || preparedLine.IndexOf('[') >= 0); if (hasTemplateIdDeclaration) { - foreach (Match match in - CppTemplateIdDeclarationRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + CppTemplateIdDeclarationRegex, + preparedLine)) { + if (line.LimitReached) + break; + if (IsCppTemplateDeclarationOrSpecializationLine( preparedLine, match.Index)) @@ -572,8 +605,13 @@ private static void EmitCppConstraintAndDeclarationReferences( if (!hasDeclarationType) return; - foreach (Match match in CppDeclarationTypeRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + CppDeclarationTypeRegex, + preparedLine)) { + if (line.LimitReached) + break; + var group = match.Groups["type"]; var expression = StripCppAccessPrefix(group.Value); if (expression.Length == 0) @@ -604,10 +642,13 @@ private static void EmitCppConceptReferences( line.EmitTypeExpressions(CppParenthesizedRequiresConceptTypeRegex); if (hasRequiresConcept && hasScopeSeparator) { - foreach (Match match in - CppQualifiedRequiresConceptConstraintRegex.Matches( + foreach (Match match in Regex.EnumerateMatches( + CppQualifiedRequiresConceptConstraintRegex, preparedLine)) { + if (line.LimitReached) + break; + line.AddTypeExpression(match.Groups["concept"]); line.AddTypeExpression(match.Groups["args"]); } diff --git a/src/CodeIndex/Indexer/References/Support/LanguageReferenceExtractionSupport.Go.cs b/src/CodeIndex/Indexer/References/Support/LanguageReferenceExtractionSupport.Go.cs index 7c7c5fd774..647dcfeffe 100644 --- a/src/CodeIndex/Indexer/References/Support/LanguageReferenceExtractionSupport.Go.cs +++ b/src/CodeIndex/Indexer/References/Support/LanguageReferenceExtractionSupport.Go.cs @@ -240,8 +240,13 @@ private static void EmitGoTypeReferences( if (preparedLine.IndexOf('{') >= 0 && ContainsGoUppercaseAscii(preparedLine)) { - foreach (Match match in GoCompositeLiteralRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + GoCompositeLiteralRegex, + preparedLine)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var group = match.Groups["name"]; if (!IsGoCompositeLiteralContext(preparedLine, group.Index, group.Value.Length)) continue; diff --git a/src/CodeIndex/Indexer/References/Support/LanguageReferenceExtractionSupport.GoCompositeTypes.cs b/src/CodeIndex/Indexer/References/Support/LanguageReferenceExtractionSupport.GoCompositeTypes.cs index 838a818b5b..e4e0a49f82 100644 --- a/src/CodeIndex/Indexer/References/Support/LanguageReferenceExtractionSupport.GoCompositeTypes.cs +++ b/src/CodeIndex/Indexer/References/Support/LanguageReferenceExtractionSupport.GoCompositeTypes.cs @@ -15,8 +15,13 @@ private static void EmitGoBuiltinTypeArgumentReferences( int lineNumber, Func resolveContainerForColumn) { - foreach (Match match in GoBuiltinTypeArgumentRegex.Matches(line)) + foreach (Match match in Regex.EnumerateMatches( + GoBuiltinTypeArgumentRegex, + line)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var open = line.IndexOf('(', match.Index); if (open < 0) continue; @@ -50,8 +55,11 @@ private static void EmitGoTypeAssertionReferences( int lineNumber, Func resolveContainerForColumn) { - foreach (Match match in GoTypeAssertionRegex.Matches(line)) + foreach (Match match in Regex.EnumerateMatches(GoTypeAssertionRegex, line)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var group = match.Groups["type"]; var expression = group.Value.Trim(); if (expression.Length == 0 || string.Equals(expression, "type", StringComparison.Ordinal)) @@ -166,8 +174,13 @@ private static void EmitGoFunctionLiteralSignatureTypes( int lineNumber, Func resolveContainerForColumn) { - foreach (Match match in GoFunctionLiteralRegex.Matches(line)) + foreach (Match match in Regex.EnumerateMatches( + GoFunctionLiteralRegex, + line)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + var open = line.IndexOf('(', match.Index); if (open < 0) continue; diff --git a/src/CodeIndex/Indexer/References/Support/LanguageReferenceExtractionSupport.GoSignatures.cs b/src/CodeIndex/Indexer/References/Support/LanguageReferenceExtractionSupport.GoSignatures.cs index 499a485fe8..a7f86e67bc 100644 --- a/src/CodeIndex/Indexer/References/Support/LanguageReferenceExtractionSupport.GoSignatures.cs +++ b/src/CodeIndex/Indexer/References/Support/LanguageReferenceExtractionSupport.GoSignatures.cs @@ -8,7 +8,9 @@ internal static partial class LanguageReferenceExtractionSupport { internal static void EmitGoBranchLabelReferences(string preparedLine, Action addCallLikeReference) { - foreach (Match match in GoBranchLabelRegex.Matches(preparedLine)) + foreach (Match match in Regex.EnumerateMatches( + GoBranchLabelRegex, + preparedLine)) addCallLikeReference(match.Groups["name"].Value, match.Groups["name"].Index); } From ba8a794ef8b9e1903f22b303ac0ec0fdacc9b184 Mon Sep 17 00:00:00 2001 From: Widthdom Date: Sun, 26 Jul 2026 18:12:26 +0900 Subject: [PATCH 18/30] Stream SQL reference matches --- DEVELOPER_GUIDE.md | 8 +++ TESTING_GUIDE.md | 2 + .../+stream-sql-reference-matches.fixed.md | 27 ++++++++++ .../SqlReferenceExtractor.AlterTargets.cs | 52 ++++++++++++------- .../SqlReferenceExtractor.DropTargets.cs | 32 ++++++------ .../SqlReferenceExtractor.LineMasking.cs | 8 +-- ...qlReferenceExtractor.MaintenanceTargets.cs | 16 +++--- .../SqlReferenceExtractor.QualifiedColumns.cs | 12 +++-- .../SqlReferenceExtractor.Sources.cs | 14 +++-- .../SqlReferenceExtractor.StatementState.cs | 35 ++++++++++--- .../SqlReferenceExtractor.Statements.cs | 10 ++-- .../Languages/SqlReferenceExtractor.cs | 52 +++++++++++-------- 12 files changed, 179 insertions(+), 89 deletions(-) create mode 100644 changelog.d/unreleased/+stream-sql-reference-matches.fixed.md diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index 4bc11c0802..23cc59f710 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -550,6 +550,10 @@ Systems-language scanners stream C/C++ construction and template groups, Rust calls and value/signature types, Swift property wrappers, Go concurrency and composite/signature types, plus shared scientific/native call groups. Preserve source-order emission and stop owned bounded lists at capacity. +SQL reference scanners stream statement, source, target, generated-column, +window-clause, procedure-call, and temporary-object matches. Helpers that accept +multiple SQL matches must keep the sequence demand-driven, and loops that emit +references must stop consuming it when the bounded list reaches capacity. All line-based symbol and reference extractors share `SourceLineSplitter`. It counts newline boundaries once, allocates the exact result array, and then materializes only the line strings that downstream scanners require; do not @@ -3589,6 +3593,10 @@ systems-language scanner は C / C++ construction と template group、Rust call value / signature type、Swift property wrapper、Go concurrency と composite / signature type、共有 scientific / native call group を逐次走査する。source-order emission を維持し、 所有する bounded list は上限で停止する。 +SQL reference scanner は statement、source、target、generated-column、window-clause、 +procedure-call、一時 object の match を逐次走査する。複数の SQL match を受け取る helper +は sequence を demand-driven のまま保ち、reference を出力する loop は bounded list の +上限到達時に消費を停止する。 line-based symbol / reference extractor はすべて `SourceLineSplitter` を共有する。 newline boundary を一度数えて exact result array を確保し、downstream scanner が必要とする line string だけを実体化する。`string.Split` による separator-index array を戻してはならない。 diff --git a/TESTING_GUIDE.md b/TESTING_GUIDE.md index 60177baa37..2992bc27f5 100644 --- a/TESTING_GUIDE.md +++ b/TESTING_GUIDE.md @@ -593,6 +593,7 @@ Use `docs/test-doc-maintenance-plan.md` before moving oversized suites or adding Python graph fixtures cover streamed decorator arguments, annotations, runtime type checks, typing factories, dataclass/framework integrations, and dynamic imports. `BoundedRegexTests.EnumerateMatches_InstanceRegex_StartsAtRequestedOffset` fixes the no-prefix-rescan contract used by decorator arguments. PHP, Ruby, R, and Perl graph fixtures cover streamed attributes/docblocks/types, DSL command targets, namespace/member/resource references, and arrow calls. Keep nested token/type enumeration cap-aware. Systems-language graph fixtures cover streamed C/C++ friend/construction/template groups, Rust macro/value/signature types, Swift wrappers, Go concurrency/composite/signature types, and shared scientific/native call groups. + SQL graph fixtures cover streamed statement/source/target, generated-column, window-clause, procedure-call, and temporary-object matches. Preserve source-order output and bounded-list early exits across SQL dialect branches. `ReferenceExtraction_CSharpNoAliasDenseReferences_StaysWithinAllocationBudget` keeps 12,000 no-alias calls from paying for alias dedupe, `CSharpAliasCompaction_DenseDuplicates_StaysWithinAllocationBudget` guards pre-sized stable compaction after alias rewrites, and `MutualRecursion_DenseRepeatedQualifiedNames_StaysWithinAllocationBudget` covers repeated qualified C#- and Python-style cycle names without per-edge normalization strings. `ReusableStatSnapshot_OnePassMaterialization_StaysWithinAllocationBudget` warms a 1,024-row typed snapshot with its exact capacity, then prevents a second candidate collection or redundant path values from returning to the default `net8.0` path. `ReferenceBatchTransactions_RepositoryScaleAtomicFileScopeEliminatesControlledSqlScopes` models 321,352 references distributed over 856 files in the repository snapshot's five/six-batch shape and fixes the control-SQL contract at 5,009 public batch transaction scopes versus zero explicit atomic-file batch scopes; keep it deterministic and allocation-light instead of inserting all modeled rows or asserting elapsed time. For reference-line window performance audits, measure identical prebuilt reference rows against one-batch and 32-batch limits in alternating order, report both elapsed time and `GC.GetAllocatedBytesForCurrentThread`, and remove the timing harness after recording the result; end-to-end `--memory-trace` rebuilds remain corroborating evidence because extraction, graph finalization, and OS page-cache variance can dominate the persistence delta. @@ -1459,6 +1460,7 @@ dotnet test --filter "FullyQualifiedName~GitHelperTests" Python graph fixture は decorator argument、annotation、runtime type check、typing factory、dataclass / framework integration、dynamic import の逐次走査を検証します。`BoundedRegexTests.EnumerateMatches_InstanceRegex_StartsAtRequestedOffset` は decorator argument が prefix を再走査しない契約を固定します。 PHP、Ruby、R、Perl の graph fixture は attribute / docblock / type、DSL command target、namespace / member / resource reference、arrow call の逐次走査を検証します。nested token / type enumeration も cap-aware のままにしてください。 systems-language graph fixture は C / C++ friend / construction / template group、Rust macro / value / signature type、Swift wrapper、Go concurrency / composite / signature type、共有 scientific / native call group の逐次走査を検証します。 + SQL graph fixture は statement / source / target、generated-column、window-clause、procedure-call、一時 object match の逐次走査を検証します。SQL dialect の分岐をまたいでも source-order output と bounded-list の早期停止を維持してください。 `ReferenceExtraction_CSharpNoAliasDenseReferences_StaysWithinAllocationBudget` は alias のない 12,000 call が alias dedupe のコストを負わないこと、`CSharpAliasCompaction_DenseDuplicates_StaysWithinAllocationBudget` は alias rewrite 後の事前 capacity 付き stable compaction、`MutualRecursion_DenseRepeatedQualifiedNames_StaysWithinAllocationBudget` は C# / Python 形式の qualified cycle name が edge ごとの正規化文字列を作らないことを保証します。 `ReusableStatSnapshot_OnePassMaterialization_StaysWithinAllocationBudget` は exact capacity を渡した1,024行の typed snapshot を warm-up し、2つ目の候補 collection や重複 path value が通常の `net8.0` 経路へ戻らないようにします。 `ReferenceBatchTransactions_RepositoryScaleAtomicFileScopeEliminatesControlledSqlScopes` は321,352 refsを自己snapshotの5/6 batch形状で856 filesへ分配し、制御SQL契約をpublic batch transaction scope 5,009回対explicit atomic-file scope 0回に固定します。全model rowを挿入したり経過時間をassertしたりせず、deterministicでallocation-lightなまま維持してください。reference-line windowの性能監査では、同一の事前構築済みreference rowを1-batch上限と32-batch上限で交互に測り、経過時間と`GC.GetAllocatedBytesForCurrentThread`の両方を報告して、結果記録後にtiming harnessを削除します。end-to-endの`--memory-trace` rebuildは、extraction・graph finalize・OS page cacheの変動が永続化差を支配し得るため、補助証拠として扱ってください。 diff --git a/changelog.d/unreleased/+stream-sql-reference-matches.fixed.md b/changelog.d/unreleased/+stream-sql-reference-matches.fixed.md new file mode 100644 index 0000000000..f6cbeddab0 --- /dev/null +++ b/changelog.d/unreleased/+stream-sql-reference-matches.fixed.md @@ -0,0 +1,27 @@ +--- +category: fixed +affected: + - src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.cs + - src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.AlterTargets.cs + - src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.DropTargets.cs + - src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.LineMasking.cs + - src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.MaintenanceTargets.cs + - src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.QualifiedColumns.cs + - src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.Sources.cs + - src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.StatementState.cs + - src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.Statements.cs + - DEVELOPER_GUIDE.md + - TESTING_GUIDE.md +--- + +## English + +- **Streamed SQL reference matches** — SQL statement, source, target, + generated-column, window-clause, procedure-call, and temporary-object + scanners now consume matches on demand and stop at bounded reference limits. + +## 日本語 + +- **SQL の reference match を逐次走査にしました** — SQL の statement、source、 + target、generated-column、window-clause、procedure-call、一時 object scanner は + match を demand-driven に消費し、bounded reference の上限で停止します。 diff --git a/src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.AlterTargets.cs b/src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.AlterTargets.cs index 12d52ae2d1..b149126bdd 100644 --- a/src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.AlterTargets.cs +++ b/src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.AlterTargets.cs @@ -22,7 +22,7 @@ private static void EmitAlterObjectTargetReferences( if (statement.IndexOf("VIEW", StringComparison.OrdinalIgnoreCase) >= 0) { EmitMultiTargetReferences( - AlterViewTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(AlterViewTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -40,7 +40,7 @@ private static void EmitAlterObjectTargetReferences( || statement.IndexOf("PROC", StringComparison.OrdinalIgnoreCase) >= 0) { EmitMultiTargetReferences( - AlterProcedureTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(AlterProcedureTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -57,7 +57,7 @@ private static void EmitAlterObjectTargetReferences( if (statement.IndexOf("FUNCTION", StringComparison.OrdinalIgnoreCase) >= 0) { EmitMultiTargetReferences( - AlterFunctionTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(AlterFunctionTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -74,7 +74,7 @@ private static void EmitAlterObjectTargetReferences( if (statement.IndexOf("TRIGGER", StringComparison.OrdinalIgnoreCase) >= 0) { EmitMultiTargetReferences( - AlterTriggerTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(AlterTriggerTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -91,7 +91,7 @@ private static void EmitAlterObjectTargetReferences( if (statement.IndexOf("SEQUENCE", StringComparison.OrdinalIgnoreCase) >= 0) { EmitMultiTargetReferences( - AlterSequenceTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(AlterSequenceTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -109,7 +109,7 @@ private static void EmitAlterObjectTargetReferences( && statement.IndexOf("POLICY", StringComparison.OrdinalIgnoreCase) >= 0) { EmitMultiTargetReferences( - AlterSecurityPolicyTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(AlterSecurityPolicyTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -127,7 +127,7 @@ private static void EmitAlterObjectTargetReferences( && statement.IndexOf("CATALOG", StringComparison.OrdinalIgnoreCase) >= 0) { EmitMultiTargetReferences( - AlterFullTextCatalogTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(AlterFullTextCatalogTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -145,7 +145,7 @@ private static void EmitAlterObjectTargetReferences( && statement.IndexOf("FUNCTION", StringComparison.OrdinalIgnoreCase) >= 0) { EmitMultiTargetReferences( - AlterPartitionFunctionTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(AlterPartitionFunctionTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -163,7 +163,7 @@ private static void EmitAlterObjectTargetReferences( && statement.IndexOf("SCHEME", StringComparison.OrdinalIgnoreCase) >= 0) { EmitMultiTargetReferences( - AlterPartitionSchemeTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(AlterPartitionSchemeTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -182,7 +182,7 @@ private static void EmitAlterObjectTargetReferences( && statement.IndexOf("COLLECTION", StringComparison.OrdinalIgnoreCase) >= 0) { EmitMultiTargetReferences( - AlterXmlSchemaCollectionTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(AlterXmlSchemaCollectionTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -199,7 +199,7 @@ private static void EmitAlterObjectTargetReferences( if (statement.IndexOf("ASSEMBLY", StringComparison.OrdinalIgnoreCase) >= 0) { EmitMultiTargetReferences( - AlterAssemblyTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(AlterAssemblyTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -217,7 +217,7 @@ private static void EmitAlterObjectTargetReferences( && statement.IndexOf("TRANSFER", StringComparison.OrdinalIgnoreCase) >= 0) { EmitMultiTargetReferences( - AlterSchemaTransferTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(AlterSchemaTransferTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -235,7 +235,7 @@ private static void EmitAlterObjectTargetReferences( && statement.IndexOf("SWITCH", StringComparison.OrdinalIgnoreCase) >= 0) { EmitMultiTargetReferences( - AlterTableSwitchTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(AlterTableSwitchTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -254,7 +254,7 @@ private static void EmitAlterObjectTargetReferences( && statement.IndexOf("HISTORY_TABLE", StringComparison.OrdinalIgnoreCase) >= 0) { EmitMultiTargetReferences( - AlterTableSystemVersioningHistoryTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(AlterTableSystemVersioningHistoryTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -430,7 +430,7 @@ private static void EmitWindowClauseColumnReferences( Func resolveContainerForCall, Func shouldIgnoreName) { - foreach (Match keywordMatch in WindowFrameKeywordRegex.Matches(statement)) + foreach (Match keywordMatch in BoundedRegex.EnumerateMatches(WindowFrameKeywordRegex, statement)) { if (keywordMatch.Index >= bodyStart && keywordMatch.Index < bodyEnd && keywordMatch.Index >= statementLineOffset) suppressedCallIndices.Add(keywordMatch.Index + statementStart - lineOffset); @@ -438,8 +438,12 @@ private static void EmitWindowClauseColumnReferences( foreach (var (start, end) in EnumerateWindowColumnListSpans(statement, bodyStart, bodyEnd)) { - foreach (Match match in WindowClauseColumnRegex.Matches(statement)) + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; + foreach (Match match in BoundedRegex.EnumerateMatches(WindowClauseColumnRegex, statement)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; var nameGroup = match.Groups["name"]; if (!nameGroup.Success || nameGroup.Index < start || nameGroup.Index >= end || nameGroup.Index < statementLineOffset) continue; @@ -570,8 +574,10 @@ private static void EmitProcedureCalls( Func shouldIgnoreName, Func shouldSuppressDefinitionCall) { - foreach (Match match in ProcCallRegex.Matches(statement)) + foreach (Match match in BoundedRegex.EnumerateMatches(ProcCallRegex, statement)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; if (IsInsideDoubleQuotedRegion(statement, match.Index)) continue; var nameGroup = match.Groups["name"]; @@ -606,8 +612,10 @@ private static void EmitSystemVariableReferences( long fileId, Func resolveContainerForCall) { - foreach (Match match in SystemVariableReferenceRegex.Matches(statement)) + foreach (Match match in BoundedRegex.EnumerateMatches(SystemVariableReferenceRegex, statement)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; if (IsInsideDoubleQuotedRegion(statement, match.Index)) continue; @@ -623,7 +631,7 @@ private static void EmitSystemVariableReferences( } private static void EmitSourceCaptureReferences( - MatchCollection matches, + IEnumerable matches, string statement, int statementStart, int statementLineOffset, @@ -647,6 +655,8 @@ private static void EmitSourceCaptureReferences( foreach (Match match in matches) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; if (IsInsideDoubleQuotedRegion(statement, match.Index)) continue; foreach (Capture capture in match.Groups["name"].Captures) @@ -687,8 +697,10 @@ private static void EmitMergeUsingReferences( Func resolveContainerForCall, Func shouldIgnoreName) { - foreach (Match match in MergeUsingSourceRegex.Matches(statement)) + foreach (Match match in BoundedRegex.EnumerateMatches(MergeUsingSourceRegex, statement)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; if (IsInsideDoubleQuotedRegion(statement, match.Index)) continue; var nameGroup = match.Groups["name"]; diff --git a/src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.DropTargets.cs b/src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.DropTargets.cs index 206d202cad..ad531b04bd 100644 --- a/src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.DropTargets.cs +++ b/src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.DropTargets.cs @@ -22,7 +22,7 @@ private static void EmitDropObjectTargetReferences( if (statement.IndexOf("SYNONYM", StringComparison.OrdinalIgnoreCase) >= 0) { EmitMultiTargetReferences( - DropSynonymTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(DropSynonymTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -39,7 +39,7 @@ private static void EmitDropObjectTargetReferences( if (statement.IndexOf("VIEW", StringComparison.OrdinalIgnoreCase) >= 0) { EmitMultiTargetReferences( - DropViewTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(DropViewTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -57,7 +57,7 @@ private static void EmitDropObjectTargetReferences( || statement.IndexOf("PROC", StringComparison.OrdinalIgnoreCase) >= 0) { EmitMultiTargetReferences( - DropProcedureTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(DropProcedureTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -74,7 +74,7 @@ private static void EmitDropObjectTargetReferences( if (statement.IndexOf("FUNCTION", StringComparison.OrdinalIgnoreCase) >= 0) { EmitMultiTargetReferences( - DropFunctionTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(DropFunctionTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -91,7 +91,7 @@ private static void EmitDropObjectTargetReferences( if (statement.IndexOf("TRIGGER", StringComparison.OrdinalIgnoreCase) >= 0) { EmitMultiTargetReferences( - DropTriggerTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(DropTriggerTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -108,7 +108,7 @@ private static void EmitDropObjectTargetReferences( if (statement.IndexOf("SEQUENCE", StringComparison.OrdinalIgnoreCase) >= 0) { EmitMultiTargetReferences( - DropSequenceTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(DropSequenceTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -125,7 +125,7 @@ private static void EmitDropObjectTargetReferences( if (statement.IndexOf("TYPE", StringComparison.OrdinalIgnoreCase) >= 0) { EmitMultiTargetReferences( - DropTypeTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(DropTypeTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -142,7 +142,7 @@ private static void EmitDropObjectTargetReferences( if (statement.IndexOf("RULE", StringComparison.OrdinalIgnoreCase) >= 0) { EmitMultiTargetReferences( - DropRuleTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(DropRuleTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -159,7 +159,7 @@ private static void EmitDropObjectTargetReferences( if (statement.IndexOf("DEFAULT", StringComparison.OrdinalIgnoreCase) >= 0) { EmitMultiTargetReferences( - DropDefaultTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(DropDefaultTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -176,7 +176,7 @@ private static void EmitDropObjectTargetReferences( if (statement.IndexOf("AGGREGATE", StringComparison.OrdinalIgnoreCase) >= 0) { EmitMultiTargetReferences( - DropAggregateTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(DropAggregateTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -194,7 +194,7 @@ private static void EmitDropObjectTargetReferences( && statement.IndexOf("POLICY", StringComparison.OrdinalIgnoreCase) >= 0) { EmitMultiTargetReferences( - DropSecurityPolicyTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(DropSecurityPolicyTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -212,7 +212,7 @@ private static void EmitDropObjectTargetReferences( && statement.IndexOf("CATALOG", StringComparison.OrdinalIgnoreCase) >= 0) { EmitMultiTargetReferences( - DropFullTextCatalogTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(DropFullTextCatalogTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -230,7 +230,7 @@ private static void EmitDropObjectTargetReferences( && statement.IndexOf("SCHEME", StringComparison.OrdinalIgnoreCase) >= 0) { EmitMultiTargetReferences( - DropPartitionSchemeTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(DropPartitionSchemeTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -248,7 +248,7 @@ private static void EmitDropObjectTargetReferences( && statement.IndexOf("FUNCTION", StringComparison.OrdinalIgnoreCase) >= 0) { EmitMultiTargetReferences( - DropPartitionFunctionTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(DropPartitionFunctionTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -267,7 +267,7 @@ private static void EmitDropObjectTargetReferences( && statement.IndexOf("COLLECTION", StringComparison.OrdinalIgnoreCase) >= 0) { EmitMultiTargetReferences( - DropXmlSchemaCollectionTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(DropXmlSchemaCollectionTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -284,7 +284,7 @@ private static void EmitDropObjectTargetReferences( if (statement.IndexOf("ASSEMBLY", StringComparison.OrdinalIgnoreCase) >= 0) { EmitMultiTargetReferences( - DropAssemblyTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(DropAssemblyTargetRegex, statement), statement, statementStart, statementLineOffset, diff --git a/src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.LineMasking.cs b/src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.LineMasking.cs index 2dd9fd3352..f7294faeee 100644 --- a/src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.LineMasking.cs +++ b/src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.LineMasking.cs @@ -217,9 +217,11 @@ private static bool ShouldTreatHashAsCommentCore(string line, int hashIndex) sourceStart++; else { - var usingMatches = UsingKeywordRegex.Matches(priorListItem); - if (usingMatches.Count > 0) - sourceStart = usingMatches[^1].Index + usingMatches[^1].Length; + Match? lastUsingMatch = null; + foreach (var usingMatch in BoundedRegex.EnumerateMatches(UsingKeywordRegex, priorListItem)) + lastUsingMatch = usingMatch; + if (lastUsingMatch != null) + sourceStart = lastUsingMatch.Index + lastUsingMatch.Length; else { sourceStart = priorListItem.LastIndexOf('#'); diff --git a/src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.MaintenanceTargets.cs b/src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.MaintenanceTargets.cs index d70070cc23..3d6d0ed706 100644 --- a/src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.MaintenanceTargets.cs +++ b/src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.MaintenanceTargets.cs @@ -25,7 +25,7 @@ private static void EmitMaintenanceTargetReferences( || statement.IndexOf("REVOKE", StringComparison.OrdinalIgnoreCase) >= 0) { EmitMultiTargetReferences( - ObjectPermissionTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(ObjectPermissionTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -42,7 +42,7 @@ private static void EmitMaintenanceTargetReferences( if (statement.IndexOf("AUTHORIZATION", StringComparison.OrdinalIgnoreCase) >= 0) { EmitMultiTargetReferences( - AlterAuthorizationObjectTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(AlterAuthorizationObjectTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -56,7 +56,7 @@ private static void EmitMaintenanceTargetReferences( shouldIgnoreName); EmitMultiTargetReferences( - AlterAuthorizationBareTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(AlterAuthorizationBareTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -73,7 +73,7 @@ private static void EmitMaintenanceTargetReferences( if (statement.IndexOf("STATISTICS", StringComparison.OrdinalIgnoreCase) >= 0) { EmitMultiTargetReferences( - UpdateStatisticsTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(UpdateStatisticsTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -87,7 +87,7 @@ private static void EmitMaintenanceTargetReferences( shouldIgnoreName); EmitMultiTargetReferences( - CreateStatisticsOnTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(CreateStatisticsOnTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -102,7 +102,7 @@ private static void EmitMaintenanceTargetReferences( suppressedCallIndices); EmitMultiTargetReferences( - DropStatisticsTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(DropStatisticsTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -143,7 +143,7 @@ private static void EmitMaintenanceTargetReferences( && statement.IndexOf("TABLE", StringComparison.OrdinalIgnoreCase) >= 0) { EmitMultiTargetReferences( - DropTableTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(DropTableTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -160,7 +160,7 @@ private static void EmitMaintenanceTargetReferences( if (statement.IndexOf("TRUNCATE", StringComparison.OrdinalIgnoreCase) >= 0) { EmitMultiTargetReferences( - TruncateTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(TruncateTargetRegex, statement), statement, statementStart, statementLineOffset, diff --git a/src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.QualifiedColumns.cs b/src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.QualifiedColumns.cs index 36d8cdac29..96ccd1b0d8 100644 --- a/src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.QualifiedColumns.cs +++ b/src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.QualifiedColumns.cs @@ -286,8 +286,10 @@ private static void EmitGeneratedColumnDependencyReferences( if (hasAsKeyword || hasGeneratedKeyword) { - foreach (Match match in GeneratedColumnExpressionStartRegex.Matches(statement)) + foreach (Match match in BoundedRegex.EnumerateMatches(GeneratedColumnExpressionStartRegex, statement)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; if (match.Index < statementLineOffset || IsInsideDoubleQuotedRegion(statement, match.Index)) continue; if (match.Value.TrimStart().StartsWith("AS", StringComparison.OrdinalIgnoreCase) @@ -326,8 +328,10 @@ private static void EmitGeneratedColumnDependencyReferences( && statement.IndexOf("VALUE", StringComparison.OrdinalIgnoreCase) >= 0 && statement.IndexOf("FOR", StringComparison.OrdinalIgnoreCase) >= 0) { - foreach (Match match in DefaultNextValueForExpressionRegex.Matches(statement)) + foreach (Match match in BoundedRegex.EnumerateMatches(DefaultNextValueForExpressionRegex, statement)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; if (match.Index < statementLineOffset || IsInsideDoubleQuotedRegion(statement, match.Index)) continue; @@ -366,8 +370,10 @@ private static void EmitSqlExpressionIdentifierDependencies( Func shouldIgnoreName) { var expression = statement[startIndex..endIndexExclusive]; - foreach (Match match in SqlExpressionIdentifierRegex.Matches(expression)) + foreach (Match match in BoundedRegex.EnumerateMatches(SqlExpressionIdentifierRegex, expression)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; var rawIndex = startIndex + match.Index; if (rawIndex < statementLineOffset || IsInsideDoubleQuotedRegion(statement, rawIndex)) continue; diff --git a/src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.Sources.cs b/src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.Sources.cs index 80c2977917..d73b48a7b5 100644 --- a/src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.Sources.cs +++ b/src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.Sources.cs @@ -70,7 +70,7 @@ private static string GetSourceReferenceKind(int index, IReadOnlyList? spans = null; - foreach (Match match in CteDefinitionRegex.Matches(statement)) + foreach (Match match in BoundedRegex.EnumerateMatches(CteDefinitionRegex, statement)) { if (IsInsideDoubleQuotedRegion(statement, match.Index)) continue; @@ -141,8 +141,10 @@ private static void EmitSelectIntoTargetReferences( Func shouldIgnoreName, HashSet? suppressedCallIndices = null) { - foreach (Match match in SelectIntoTargetStatementRegex.Matches(statement)) + foreach (Match match in BoundedRegex.EnumerateMatches(SelectIntoTargetStatementRegex, statement)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; if (IsInsideDoubleQuotedRegion(statement, match.Index)) continue; var nameGroup = match.Groups["name"]; @@ -172,8 +174,10 @@ private static void EmitTargetReferences( Func resolveContainerForCall, Func shouldIgnoreName) { - foreach (Match match in TargetReferenceRegex.Matches(statement)) + foreach (Match match in BoundedRegex.EnumerateMatches(TargetReferenceRegex, statement)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; if (IsInsideDoubleQuotedRegion(statement, match.Index)) continue; if (!TryGetTrailingQualifiedIdentifierLeaf(match, out var rawName, out var rawIndex)) @@ -323,7 +327,7 @@ or UnicodeCategory.SpacingCombiningMark } private static void EmitMultiTargetReferences( - MatchCollection matches, + IEnumerable matches, string statement, int statementStart, int statementLineOffset, @@ -339,6 +343,8 @@ private static void EmitMultiTargetReferences( { foreach (Match match in matches) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; if (IsInsideDoubleQuotedRegion(statement, match.Index)) continue; diff --git a/src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.StatementState.cs b/src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.StatementState.cs index feda404d46..510223184e 100644 --- a/src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.StatementState.cs +++ b/src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.StatementState.cs @@ -414,30 +414,48 @@ private static void CollectTempObjectNamesFromStatement( || statement.IndexOf("ALTER", StringComparison.OrdinalIgnoreCase) >= 0 || statement.IndexOf("BULK", StringComparison.OrdinalIgnoreCase) >= 0; if (mayContainTargetStatement) - CollectTempObjectNamesFromTargetMatches(TargetReferenceRegex.Matches(statement), statement, names); + CollectTempObjectNamesFromTargetMatches( + BoundedRegex.EnumerateMatches(TargetReferenceRegex, statement), + statement, + names); if (statement.IndexOf("TRUNCATE", StringComparison.OrdinalIgnoreCase) >= 0) - CollectTempObjectNamesFromMatches(TruncateTargetRegex.Matches(statement), statement, names); + CollectTempObjectNamesFromMatches( + BoundedRegex.EnumerateMatches(TruncateTargetRegex, statement), + statement, + names); if (statement.IndexOf("SELECT", StringComparison.OrdinalIgnoreCase) >= 0 && statement.IndexOf("INTO", StringComparison.OrdinalIgnoreCase) >= 0) { - CollectTempObjectNamesFromMatches(SelectIntoTargetStatementRegex.Matches(statement), statement, names); + CollectTempObjectNamesFromMatches( + BoundedRegex.EnumerateMatches(SelectIntoTargetStatementRegex, statement), + statement, + names); } if (statement.IndexOf("CREATE", StringComparison.OrdinalIgnoreCase) < 0) return; if (statement.IndexOf("TABLE", StringComparison.OrdinalIgnoreCase) >= 0) - CollectTempObjectNamesFromMatches(CreateTempTableRegex.Matches(statement), statement, names); + CollectTempObjectNamesFromMatches( + BoundedRegex.EnumerateMatches(CreateTempTableRegex, statement), + statement, + names); if (statement.IndexOf("PROC", StringComparison.OrdinalIgnoreCase) >= 0 || statement.IndexOf("FUNCTION", StringComparison.OrdinalIgnoreCase) >= 0) { - CollectTempObjectNamesFromMatches(CreateTempRoutineRegex.Matches(statement), statement, names); + CollectTempObjectNamesFromMatches( + BoundedRegex.EnumerateMatches(CreateTempRoutineRegex, statement), + statement, + names); } } - private static void CollectTempObjectNamesFromTargetMatches(MatchCollection matches, string statement, HashSet names) + private static void CollectTempObjectNamesFromTargetMatches( + IEnumerable matches, + string statement, + HashSet names) { foreach (Match match in matches) { @@ -452,7 +470,10 @@ private static void CollectTempObjectNamesFromTargetMatches(MatchCollection matc } } - private static void CollectTempObjectNamesFromMatches(MatchCollection matches, string statement, HashSet names) + private static void CollectTempObjectNamesFromMatches( + IEnumerable matches, + string statement, + HashSet names) { foreach (Match match in matches) { diff --git a/src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.Statements.cs b/src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.Statements.cs index 0a05202689..0b26b9b685 100644 --- a/src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.Statements.cs +++ b/src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.Statements.cs @@ -107,7 +107,7 @@ private static void EmitStatementReferences( if (hasFromKeyword) { EmitSourceCaptureReferences( - FromSourceListRegex.Matches(statement), + BoundedRegex.EnumerateMatches(FromSourceListRegex, statement), statement, statementStart, statementLineOffset, @@ -127,7 +127,7 @@ private static void EmitStatementReferences( if (hasJoinKeyword || hasApplyKeyword) { EmitSourceCaptureReferences( - SourceReferenceRegex.Matches(statement), + BoundedRegex.EnumerateMatches(SourceReferenceRegex, statement), statement, statementStart, statementLineOffset, @@ -167,7 +167,7 @@ private static void EmitStatementReferences( if (hasUsingKeyword) { EmitSourceCaptureReferences( - DeleteUsingSourceRegex.Matches(statement), + BoundedRegex.EnumerateMatches(DeleteUsingSourceRegex, statement), statement, statementStart, statementLineOffset, @@ -184,7 +184,7 @@ private static void EmitStatementReferences( } EmitMultiTargetReferences( - DeleteTargetWithoutFromRegex.Matches(statement), + BoundedRegex.EnumerateMatches(DeleteTargetWithoutFromRegex, statement), statement, statementStart, statementLineOffset, @@ -201,7 +201,7 @@ private static void EmitStatementReferences( if (hasOutputKeyword && hasIntoKeyword) { EmitMultiTargetReferences( - OutputIntoTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(OutputIntoTargetRegex, statement), statement, statementStart, statementLineOffset, diff --git a/src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.cs b/src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.cs index e27009707a..19676d432e 100644 --- a/src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.cs +++ b/src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.cs @@ -204,7 +204,7 @@ public static HashSet Emit( HashSet? usingSourceIndices = null; if (hasMergeKeyword && hasUsingKeyword) { - foreach (Match match in MergeUsingSourceRegex.Matches(statement)) + foreach (Match match in BoundedRegex.EnumerateMatches(MergeUsingSourceRegex, statement)) { if (IsInsideDoubleQuotedRegion(statement, match.Index)) continue; @@ -218,7 +218,7 @@ public static HashSet Emit( if (hasDeleteKeyword && hasUsingKeyword) { - foreach (Match match in DeleteUsingSourceRegex.Matches(statement)) + foreach (Match match in BoundedRegex.EnumerateMatches(DeleteUsingSourceRegex, statement)) { if (IsInsideDoubleQuotedRegion(statement, match.Index)) continue; @@ -235,7 +235,7 @@ public static HashSet Emit( if (statement.IndexOf("TOP", StringComparison.OrdinalIgnoreCase) >= 0) { - foreach (Match match in TopCallSuppressionRegex.Matches(statement)) + foreach (Match match in BoundedRegex.EnumerateMatches(TopCallSuppressionRegex, statement)) { var nameGroup = match.Groups["name"]; if (nameGroup.Index < statementLineOffset) @@ -249,7 +249,7 @@ public static HashSet Emit( && statement.IndexOf("INDEX", StringComparison.OrdinalIgnoreCase) >= 0 && hasUsingKeyword) { - foreach (Match match in AccessMethodCallSuppressionRegex.Matches(statement)) + foreach (Match match in BoundedRegex.EnumerateMatches(AccessMethodCallSuppressionRegex, statement)) { if (IsInsideDoubleQuotedRegion(statement, match.Index)) continue; @@ -336,7 +336,7 @@ private static void EmitCreateIndexTargetReferences( return; EmitMultiTargetReferences( - CreateIndexOnTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(CreateIndexOnTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -353,7 +353,7 @@ private static void EmitCreateIndexTargetReferences( if (statement.IndexOf("XML", StringComparison.OrdinalIgnoreCase) >= 0) { EmitMultiTargetReferences( - CreateSpecialXmlIndexOnTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(CreateSpecialXmlIndexOnTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -371,7 +371,7 @@ private static void EmitCreateIndexTargetReferences( if (statement.IndexOf("COLUMNSTORE", StringComparison.OrdinalIgnoreCase) >= 0) { EmitMultiTargetReferences( - CreateClusteredColumnstoreIndexOnTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(CreateClusteredColumnstoreIndexOnTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -389,7 +389,7 @@ private static void EmitCreateIndexTargetReferences( if (statement.IndexOf("HASH", StringComparison.OrdinalIgnoreCase) >= 0) { EmitMultiTargetReferences( - CreateHashIndexOnTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(CreateHashIndexOnTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -407,7 +407,7 @@ private static void EmitCreateIndexTargetReferences( if (statement.IndexOf("FULLTEXT", StringComparison.OrdinalIgnoreCase) >= 0) { EmitMultiTargetReferences( - CreateFullTextIndexOnTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(CreateFullTextIndexOnTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -440,7 +440,7 @@ private static void EmitAlterAndDropIndexTargetReferences( && statement.IndexOf("ON", StringComparison.OrdinalIgnoreCase) >= 0) { EmitMultiTargetReferences( - AlterIndexOnTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(AlterIndexOnTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -456,7 +456,7 @@ private static void EmitAlterAndDropIndexTargetReferences( if (statement.IndexOf("FULLTEXT", StringComparison.OrdinalIgnoreCase) >= 0) { EmitMultiTargetReferences( - AlterFullTextIndexOnTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(AlterFullTextIndexOnTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -477,7 +477,7 @@ private static void EmitAlterAndDropIndexTargetReferences( if (statement.IndexOf("ON", StringComparison.OrdinalIgnoreCase) >= 0) { EmitMultiTargetReferences( - DropIndexOnTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(DropIndexOnTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -492,7 +492,7 @@ private static void EmitAlterAndDropIndexTargetReferences( } EmitMultiTargetReferences( - DropIndexLegacyTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(DropIndexLegacyTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -509,7 +509,7 @@ private static void EmitAlterAndDropIndexTargetReferences( && statement.IndexOf("ON", StringComparison.OrdinalIgnoreCase) >= 0) { EmitMultiTargetReferences( - DropFullTextIndexOnTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(DropFullTextIndexOnTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -550,7 +550,7 @@ private static void EmitObjectLifecycleTargetReferences( if (hasCreateKeyword && hasTriggerKeyword && hasOnKeyword) { EmitMultiTargetReferences( - CreateTriggerOnTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(CreateTriggerOnTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -571,7 +571,7 @@ private static void EmitObjectLifecycleTargetReferences( && hasOnKeyword) { EmitMultiTargetReferences( - CreateSecurityPolicyPredicateTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(CreateSecurityPolicyPredicateTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -592,7 +592,7 @@ private static void EmitObjectLifecycleTargetReferences( && hasOnKeyword) { EmitMultiTargetReferences( - AlterSecurityPolicyPredicateTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(AlterSecurityPolicyPredicateTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -612,7 +612,7 @@ private static void EmitObjectLifecycleTargetReferences( && hasOnKeyword) { EmitMultiTargetReferences( - ToggleTriggerOnTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(ToggleTriggerOnTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -629,7 +629,7 @@ private static void EmitObjectLifecycleTargetReferences( if (statement.IndexOf("REFERENCES", StringComparison.OrdinalIgnoreCase) >= 0) { EmitMultiTargetReferences( - ForeignKeyReferencesTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(ForeignKeyReferencesTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -649,7 +649,7 @@ private static void EmitObjectLifecycleTargetReferences( && statement.IndexOf("FOR", StringComparison.OrdinalIgnoreCase) >= 0) { EmitMultiTargetReferences( - CreateSynonymForTargetRegex.Matches(statement), + BoundedRegex.EnumerateMatches(CreateSynonymForTargetRegex, statement), statement, statementStart, statementLineOffset, @@ -693,8 +693,10 @@ private static void EmitMergeOnColumnReferences( Func resolveContainerForCall, Func shouldIgnoreName) { - foreach (Match match in MergeOnClauseRegex.Matches(statement)) + foreach (Match match in BoundedRegex.EnumerateMatches(MergeOnClauseRegex, statement)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; if (IsInsideDoubleQuotedRegion(statement, match.Index)) continue; @@ -730,8 +732,10 @@ private static void EmitMergeUpdateColumnReferences( Func resolveContainerForCall, Func shouldIgnoreName) { - foreach (Match match in MergeUpdateSetActionRegex.Matches(statement)) + foreach (Match match in BoundedRegex.EnumerateMatches(MergeUpdateSetActionRegex, statement)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; if (IsInsideDoubleQuotedRegion(statement, match.Index)) continue; @@ -790,8 +794,10 @@ private static void EmitMergeInsertColumnReferences( Func resolveContainerForCall, Func shouldIgnoreName) { - foreach (Match match in MergeInsertActionRegex.Matches(statement)) + foreach (Match match in BoundedRegex.EnumerateMatches(MergeInsertActionRegex, statement)) { + if (ReferenceExtractor.ReferenceLimitReached(references)) + break; if (IsInsideDoubleQuotedRegion(statement, match.Index)) continue; From 85d328668f43d64ab55aef3717db4e442339bc1c Mon Sep 17 00:00:00 2001 From: Widthdom Date: Sun, 26 Jul 2026 18:21:50 +0900 Subject: [PATCH 19/30] Stream infrastructure reference matches --- DEVELOPER_GUIDE.md | 8 ++++ TESTING_GUIDE.md | 2 + ...-infrastructure-reference-matches.fixed.md | 27 +++++++++++ .../BuildAutomationReferenceExtractor.cs | 8 +++- .../Languages/CssReferenceExtractor.cs | 16 +++++-- .../Languages/DockerfileReferenceExtractor.cs | 14 ++++-- .../Languages/HdlReferenceExtractor.Scopes.cs | 4 +- .../Languages/HdlReferenceExtractor.cs | 26 +++++++--- .../MarkupSchemaReferenceExtractor.cs | 48 +++++++++++++++---- .../Languages/PowerShellReferenceExtractor.cs | 8 ++-- .../Languages/ShellReferenceExtractor.cs | 12 +++-- .../Languages/XamlReferenceExtractor.cs | 30 ++++++++---- 12 files changed, 160 insertions(+), 43 deletions(-) create mode 100644 changelog.d/unreleased/+stream-infrastructure-reference-matches.fixed.md diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index 23cc59f710..cb066eb532 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -554,6 +554,10 @@ SQL reference scanners stream statement, source, target, generated-column, window-clause, procedure-call, and temporary-object matches. Helpers that accept multiple SQL matches must keep the sequence demand-driven, and loops that emit references must stop consuming it when the bounded list reaches capacity. +Infrastructure and markup scanners stream CSS, XAML, HTML/GraphQL/Markdown, +HDL, MSBuild, Dockerfile, shell, and PowerShell match groups. Keep state-only +scans demand-driven as well, while applying bounded-list exits only where the +scanner owns the reference list. All line-based symbol and reference extractors share `SourceLineSplitter`. It counts newline boundaries once, allocates the exact result array, and then materializes only the line strings that downstream scanners require; do not @@ -3597,6 +3601,10 @@ SQL reference scanner は statement、source、target、generated-column、windo procedure-call、一時 object の match を逐次走査する。複数の SQL match を受け取る helper は sequence を demand-driven のまま保ち、reference を出力する loop は bounded list の 上限到達時に消費を停止する。 +infrastructure / markup scanner は CSS、XAML、HTML / GraphQL / Markdown、HDL、 +MSBuild、Dockerfile、shell、PowerShell の match group を逐次走査する。state-only scan +も demand-driven のまま保ち、bounded-list の停止判定は scanner が reference list を +所有する箇所だけに適用する。 line-based symbol / reference extractor はすべて `SourceLineSplitter` を共有する。 newline boundary を一度数えて exact result array を確保し、downstream scanner が必要とする line string だけを実体化する。`string.Split` による separator-index array を戻してはならない。 diff --git a/TESTING_GUIDE.md b/TESTING_GUIDE.md index 2992bc27f5..dcf27c30aa 100644 --- a/TESTING_GUIDE.md +++ b/TESTING_GUIDE.md @@ -594,6 +594,7 @@ Use `docs/test-doc-maintenance-plan.md` before moving oversized suites or adding PHP, Ruby, R, and Perl graph fixtures cover streamed attributes/docblocks/types, DSL command targets, namespace/member/resource references, and arrow calls. Keep nested token/type enumeration cap-aware. Systems-language graph fixtures cover streamed C/C++ friend/construction/template groups, Rust macro/value/signature types, Swift wrappers, Go concurrency/composite/signature types, and shared scientific/native call groups. SQL graph fixtures cover streamed statement/source/target, generated-column, window-clause, procedure-call, and temporary-object matches. Preserve source-order output and bounded-list early exits across SQL dialect branches. + Infrastructure and markup graph fixtures cover streamed CSS, XAML, HTML/GraphQL/Markdown, HDL, MSBuild, Dockerfile, shell, and PowerShell matches. Keep state-building scans independent from bounded reference-list ownership. `ReferenceExtraction_CSharpNoAliasDenseReferences_StaysWithinAllocationBudget` keeps 12,000 no-alias calls from paying for alias dedupe, `CSharpAliasCompaction_DenseDuplicates_StaysWithinAllocationBudget` guards pre-sized stable compaction after alias rewrites, and `MutualRecursion_DenseRepeatedQualifiedNames_StaysWithinAllocationBudget` covers repeated qualified C#- and Python-style cycle names without per-edge normalization strings. `ReusableStatSnapshot_OnePassMaterialization_StaysWithinAllocationBudget` warms a 1,024-row typed snapshot with its exact capacity, then prevents a second candidate collection or redundant path values from returning to the default `net8.0` path. `ReferenceBatchTransactions_RepositoryScaleAtomicFileScopeEliminatesControlledSqlScopes` models 321,352 references distributed over 856 files in the repository snapshot's five/six-batch shape and fixes the control-SQL contract at 5,009 public batch transaction scopes versus zero explicit atomic-file batch scopes; keep it deterministic and allocation-light instead of inserting all modeled rows or asserting elapsed time. For reference-line window performance audits, measure identical prebuilt reference rows against one-batch and 32-batch limits in alternating order, report both elapsed time and `GC.GetAllocatedBytesForCurrentThread`, and remove the timing harness after recording the result; end-to-end `--memory-trace` rebuilds remain corroborating evidence because extraction, graph finalization, and OS page-cache variance can dominate the persistence delta. @@ -1461,6 +1462,7 @@ dotnet test --filter "FullyQualifiedName~GitHelperTests" PHP、Ruby、R、Perl の graph fixture は attribute / docblock / type、DSL command target、namespace / member / resource reference、arrow call の逐次走査を検証します。nested token / type enumeration も cap-aware のままにしてください。 systems-language graph fixture は C / C++ friend / construction / template group、Rust macro / value / signature type、Swift wrapper、Go concurrency / composite / signature type、共有 scientific / native call group の逐次走査を検証します。 SQL graph fixture は statement / source / target、generated-column、window-clause、procedure-call、一時 object match の逐次走査を検証します。SQL dialect の分岐をまたいでも source-order output と bounded-list の早期停止を維持してください。 + infrastructure / markup graph fixture は CSS、XAML、HTML / GraphQL / Markdown、HDL、MSBuild、Dockerfile、shell、PowerShell match の逐次走査を検証します。state 構築 scan は bounded reference-list の所有権から独立したままにしてください。 `ReferenceExtraction_CSharpNoAliasDenseReferences_StaysWithinAllocationBudget` は alias のない 12,000 call が alias dedupe のコストを負わないこと、`CSharpAliasCompaction_DenseDuplicates_StaysWithinAllocationBudget` は alias rewrite 後の事前 capacity 付き stable compaction、`MutualRecursion_DenseRepeatedQualifiedNames_StaysWithinAllocationBudget` は C# / Python 形式の qualified cycle name が edge ごとの正規化文字列を作らないことを保証します。 `ReusableStatSnapshot_OnePassMaterialization_StaysWithinAllocationBudget` は exact capacity を渡した1,024行の typed snapshot を warm-up し、2つ目の候補 collection や重複 path value が通常の `net8.0` 経路へ戻らないようにします。 `ReferenceBatchTransactions_RepositoryScaleAtomicFileScopeEliminatesControlledSqlScopes` は321,352 refsを自己snapshotの5/6 batch形状で856 filesへ分配し、制御SQL契約をpublic batch transaction scope 5,009回対explicit atomic-file scope 0回に固定します。全model rowを挿入したり経過時間をassertしたりせず、deterministicでallocation-lightなまま維持してください。reference-line windowの性能監査では、同一の事前構築済みreference rowを1-batch上限と32-batch上限で交互に測り、経過時間と`GC.GetAllocatedBytesForCurrentThread`の両方を報告して、結果記録後にtiming harnessを削除します。end-to-endの`--memory-trace` rebuildは、extraction・graph finalize・OS page cacheの変動が永続化差を支配し得るため、補助証拠として扱ってください。 diff --git a/changelog.d/unreleased/+stream-infrastructure-reference-matches.fixed.md b/changelog.d/unreleased/+stream-infrastructure-reference-matches.fixed.md new file mode 100644 index 0000000000..3c7732614e --- /dev/null +++ b/changelog.d/unreleased/+stream-infrastructure-reference-matches.fixed.md @@ -0,0 +1,27 @@ +--- +category: fixed +affected: + - src/CodeIndex/Indexer/References/Languages/BuildAutomationReferenceExtractor.cs + - src/CodeIndex/Indexer/References/Languages/CssReferenceExtractor.cs + - src/CodeIndex/Indexer/References/Languages/DockerfileReferenceExtractor.cs + - src/CodeIndex/Indexer/References/Languages/HdlReferenceExtractor.cs + - src/CodeIndex/Indexer/References/Languages/HdlReferenceExtractor.Scopes.cs + - src/CodeIndex/Indexer/References/Languages/MarkupSchemaReferenceExtractor.cs + - src/CodeIndex/Indexer/References/Languages/PowerShellReferenceExtractor.cs + - src/CodeIndex/Indexer/References/Languages/ShellReferenceExtractor.cs + - src/CodeIndex/Indexer/References/Languages/XamlReferenceExtractor.cs + - DEVELOPER_GUIDE.md + - TESTING_GUIDE.md +--- + +## English + +- **Streamed infrastructure and markup reference matches** — CSS, XAML, + HTML/GraphQL/Markdown, HDL, MSBuild, Dockerfile, shell, and PowerShell + scanners now consume dense match groups on demand. + +## 日本語 + +- **infrastructure / markup の reference match を逐次走査にしました** — CSS、 + XAML、HTML / GraphQL / Markdown、HDL、MSBuild、Dockerfile、shell、PowerShell + scanner は dense な match group を demand-driven に消費します。 diff --git a/src/CodeIndex/Indexer/References/Languages/BuildAutomationReferenceExtractor.cs b/src/CodeIndex/Indexer/References/Languages/BuildAutomationReferenceExtractor.cs index 2ce0e59d2d..8430f25ccb 100644 --- a/src/CodeIndex/Indexer/References/Languages/BuildAutomationReferenceExtractor.cs +++ b/src/CodeIndex/Indexer/References/Languages/BuildAutomationReferenceExtractor.cs @@ -213,15 +213,19 @@ private static void EmitMsBuildReferences( if (line.TrimStart().StartsWith("