Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
3d7715a
Reduce masked-line reference allocations
Widthdom Jul 26, 2026
04ec0b2
Scan C++ header samples without line arrays
Widthdom Jul 26, 2026
ba009a9
Walk delimited extractor inputs with spans
Widthdom Jul 26, 2026
55ec874
Avoid functional span predicate closures
Widthdom Jul 26, 2026
35d8fc0
Avoid hardware scope predicate closures
Widthdom Jul 26, 2026
5c04709
Stream metadata character and ancestor checks
Widthdom Jul 26, 2026
6e03004
Split source lines at exact capacity
Widthdom Jul 26, 2026
49e90df
Reuse prepared-line whitespace classification
Widthdom Jul 26, 2026
9ddf12d
Compare functional terminators on spans
Widthdom Jul 26, 2026
299854e
Share trimmed suffix checks across scanners
Widthdom Jul 26, 2026
3f41d83
Allocate functional exclusion spans lazily
Widthdom Jul 26, 2026
14d88e8
Stream bounded regex match enumeration
Widthdom Jul 26, 2026
74b182a
Stream functional reference matches
Widthdom Jul 26, 2026
b2ce2f2
Stream JVM reference matches
Widthdom Jul 26, 2026
cf9d7eb
Stream Python reference matches
Widthdom Jul 26, 2026
a96f13c
Stream dynamic language reference matches
Widthdom Jul 26, 2026
0ae9365
Stream systems language reference matches
Widthdom Jul 26, 2026
ba8a794
Stream SQL reference matches
Widthdom Jul 26, 2026
85d3286
Stream infrastructure reference matches
Widthdom Jul 26, 2026
1cebf5b
Stream core reference matches
Widthdom Jul 26, 2026
58a9207
Preserve right-to-left regex enumeration
Widthdom Jul 26, 2026
f24882c
Stream secondary reference matches
Widthdom Jul 26, 2026
dd3f308
Stop capped reference scans immediately
Widthdom Jul 26, 2026
6e52d7f
Stream symbol and dependency regex matches
Widthdom Jul 26, 2026
1c524ea
Stop XAML symbol scans at the shared budget
Widthdom Jul 26, 2026
1120878
Reuse symbol container path buffers
Widthdom Jul 26, 2026
4d19180
Avoid empty Prolog goal lists
Widthdom Jul 26, 2026
4c15d16
Build JSON line offsets at exact capacity
Widthdom Jul 26, 2026
4919a6e
Stop bounded match scans before advancing
Widthdom Jul 26, 2026
4e16765
Make Prolog allocation fixture platform-neutral
Widthdom Jul 26, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
198 changes: 196 additions & 2 deletions DEVELOPER_GUIDE.md

Large diffs are not rendered by default.

54 changes: 54 additions & 0 deletions TESTING_GUIDE.md

Large diffs are not rendered by default.

20 changes: 20 additions & 0 deletions changelog.d/unreleased/+cpp-header-line-walk.fixed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
---
category: fixed
affected:
- src/CodeIndex/Indexer/Scanning/FileIndexer.LanguageDetection.cs
- tests/CodeIndex.Tests/PerformanceTests.cs
- DEVELOPER_GUIDE.md
- TESTING_GUIDE.md
---

## English

- **Reduced C / C++ header-detection allocations for large repositories** —
Bounded lexical samples for ambiguous `.h` files are now scanned by newline
index instead of materializing an array and a string for every sampled line.

## 日本語

- **巨大 repository の C / C++ header 判定 allocation を削減しました** —
曖昧な `.h` file の bounded lexical sample を、sampled line ごとの array と string
に実体化せず newline index で走査するようにしました。
27 changes: 27 additions & 0 deletions changelog.d/unreleased/+delimited-extractor-spans.fixed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
---
category: fixed
affected:
- src/CodeIndex/Indexer/DelimitedSpanEnumerable.cs
- src/CodeIndex/Indexer/Symbols/SymbolExtractor.RepositoryMetadata.cs
- src/CodeIndex/Indexer/References/Languages/RepositoryMetadataReferenceExtractor.cs
- src/CodeIndex/Indexer/References/Languages/HdlReferenceExtractor.cs
- src/CodeIndex/Indexer/References/Languages/HdlReferenceExtractor.Scopes.cs
- src/CodeIndex/Indexer/References/Languages/ShaderReferenceExtractor.cs
- tests/CodeIndex.Tests/PerformanceTests.cs
- DEVELOPER_GUIDE.md
- TESTING_GUIDE.md
---

## English

- **Removed split-array growth from delimiter-only extractor paths** —
Repository metadata, application manifests, VHDL declarations and package
imports, and CUDA kernel parameters now share an allocation-free span walker
while preserving trimming and empty-entry behavior.

## 日本語

- **delimiter-only extractor 経路の split-array 増加を解消しました** —
repository metadata、application manifest、VHDL declaration / package import、
CUDA kernel parameter は、trim と empty-entry の意味論を維持しつつ allocation-free
な span walker を共有します。
23 changes: 23 additions & 0 deletions changelog.d/unreleased/+functional-span-membership.fixed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
---
category: fixed
affected:
- src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.cs
- src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Erlang.cs
- src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Ocaml.cs
- src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Raku.cs
- tests/CodeIndex.Tests/PerformanceTests.cs
- DEVELOPER_GUIDE.md
- TESTING_GUIDE.md
---

## English

- **Removed per-match closure allocations from functional-language references** —
Erlang, OCAML, and Raku now use shared indexed span-membership checks when
suppressing remote, qualified, quoted-atom, and type-reference matches.

## 日本語

- **functional-language reference の match ごとの closure allocation を解消しました** —
Erlang、OCAML、Raku は remote、qualified、quoted-atom、type-reference match の抑制時に、
共通の indexed span-membership check を使います。
21 changes: 21 additions & 0 deletions changelog.d/unreleased/+functional-terminator-spans.fixed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
---
category: fixed
affected:
- src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.cs
- src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Erlang.cs
- tests/CodeIndex.Tests/PerformanceTests.cs
- DEVELOPER_GUIDE.md
- TESTING_GUIDE.md
---

## English

- **Removed padded-line copies from functional-language state machines** —
Erlang specification/callable terminators and Raku heredoc terminators now
compare trimmed spans, avoiding a new string for every padded sentinel line.

## 日本語

- **functional-language state machine の padded-line copy を解消しました** —
Erlang specification / callable terminator と Raku heredoc terminator は trimmed span
を比較し、padding 付き sentinel line ごとの新しい string を回避します。
21 changes: 21 additions & 0 deletions changelog.d/unreleased/+hardware-scope-membership.fixed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
---
category: fixed
affected:
- src/CodeIndex/Indexer/References/Languages/HdlReferenceExtractor.cs
- src/CodeIndex/Indexer/References/Languages/ShaderReferenceExtractor.cs
- tests/CodeIndex.Tests/PerformanceTests.cs
- DEVELOPER_GUIDE.md
- TESTING_GUIDE.md
---

## English

- **Removed per-identifier closure allocations from hardware-language scopes** —
Verilog, SystemVerilog, and VHDL shadow checks plus CUDA, GLSL, HLSL, Metal,
and WGSL binding/resource checks now use direct indexed membership loops.

## 日本語

- **hardware-language scope の identifier ごとの closure allocation を解消しました** —
Verilog、SystemVerilog、VHDL の shadow 判定と、CUDA、GLSL、HLSL、Metal、WGSL
の binding / resource 判定は direct indexed membership loop を使います。
23 changes: 23 additions & 0 deletions changelog.d/unreleased/+lazy-functional-spans.fixed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
---
category: fixed
affected:
- src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.cs
- src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Erlang.cs
- src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Ocaml.cs
- src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Raku.cs
- tests/CodeIndex.Tests/PerformanceTests.cs
- DEVELOPER_GUIDE.md
- TESTING_GUIDE.md
---

## English

- **Stopped allocating empty exclusion lists on functional-language lines** —
Erlang, OCAML, and Raku now create quoted, remote, type, qualified, and method
span lists only after the first relevant match on a line.

## 日本語

- **functional-language line ごとの empty exclusion list allocation を解消しました** —
Erlang、OCAML、Raku は quoted、remote、type、qualified、method span list を、行内の
最初の関連 match が見つかった後にだけ作ります。
21 changes: 21 additions & 0 deletions changelog.d/unreleased/+masked-reference-contexts.fixed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
---
category: fixed
affected:
- src/CodeIndex/Indexer/References/ReferenceExtractor.CoreReferenceLoop.cs
- tests/CodeIndex.Tests/PerformanceTests.cs
- DEVELOPER_GUIDE.md
- TESTING_GUIDE.md
---

## English

- **Reduced reference-extraction allocations for large embedded multiline payloads** —
Structurally masked C# raw strings, Java text blocks, and JavaScript /
TypeScript template literals no longer materialize trimmed reference
contexts for lines that will be skipped.

## 日本語

- **巨大な埋め込み multiline payload に対する reference extraction の allocation を削減しました** —
構造マスク済みの C# raw string、Java text block、JavaScript / TypeScript template literal
では、skip される行の trim 済み reference context を実体化しません。
24 changes: 24 additions & 0 deletions changelog.d/unreleased/+metadata-character-scans.fixed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
---
category: fixed
affected:
- src/CodeIndex/Indexer/SpanCharacterSearch.cs
- src/CodeIndex/Indexer/Symbols/SymbolExtractor.RepositoryMetadata.cs
- src/CodeIndex/Indexer/References/Languages/RepositoryMetadataReferenceExtractor.cs
- tests/CodeIndex.Tests/PerformanceTests.cs
- DEVELOPER_GUIDE.md
- TESTING_GUIDE.md
---

## English

- **Reduced repository-metadata and manifest validation overhead** —
Long metadata candidates now use allocation-free span character scans, while
application manifests track dependency ancestry by XML depth instead of
rescanning an ancestor stack for every identity.

## 日本語

- **repository metadata と manifest の validation overhead を削減しました** —
長い metadata candidate は allocation-free な span character scan を使い、
application manifest は identity ごとの ancestor stack 再走査ではなく XML depth
で dependency ancestry を追跡します。
19 changes: 19 additions & 0 deletions changelog.d/unreleased/+prepared-line-whitespace.fixed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
---
category: fixed
affected:
- src/CodeIndex/Indexer/References/ReferenceExtractor.CoreReferenceLoop.cs
- DEVELOPER_GUIDE.md
---

## English

- **Avoided duplicate full-line whitespace scans during reference extraction** —
Every supported language now classifies each prepared line once and reuses
the result across special-line and ordinary empty-line dispatch, which is
especially important for long structurally masked payloads.

## 日本語

- **reference extraction の重複した全行 whitespace scan を解消しました** —
全対応言語で prepared line を一度だけ判定し、special-line と通常の empty-line dispatch
で結果を共有するため、長い構造マスク済み payload の再走査を避けます。
22 changes: 22 additions & 0 deletions changelog.d/unreleased/+source-line-splitting.fixed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
---
category: fixed
affected:
- src/CodeIndex/Indexer/SourceLineSplitter.cs
- src/CodeIndex/Indexer/Symbols/SymbolExtractor.cs
- src/CodeIndex/Indexer/References/ReferenceExtractor.Configuration.cs
- tests/CodeIndex.Tests/PerformanceTests.cs
- DEVELOPER_GUIDE.md
- TESTING_GUIDE.md
---

## English

- **Reduced all-language extraction allocations for large source files** —
Symbol and reference extraction now share an exact-capacity line splitter
that avoids the temporary separator-index arrays created by generic splitting.

## 日本語

- **巨大 source file の全言語 extraction allocation を削減しました** —
symbol / reference extraction は exact-capacity line splitter を共有し、generic split
が作る一時 separator-index array を回避します。
58 changes: 58 additions & 0 deletions changelog.d/unreleased/+stream-bounded-regex-enumeration.fixed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
---
category: fixed
affected:
- src/CodeIndex/Indexer/BoundedRegex.cs
- src/CodeIndex/Indexer/References/ReferenceExtractor.cs
- tests/CodeIndex.Tests/BoundedRegexTests.cs
- tests/CodeIndex.Tests/PerformanceTests.cs
- DEVELOPER_GUIDE.md
- TESTING_GUIDE.md
---

## English

- **Made bounded regex enumeration demand-driven** — Extractors that stop after
reaching a result limit no longer force every remaining regex match on the
source line to be materialized first, while right-to-left instances preserve
their default reverse match order. Secondary reference scanners now use the
same streaming path across Fortran, Visual Basic, F#, Pascal, Objective-C,
Haskell, Elixir, Smalltalk, Lua, Dart, Razor, JSON, JavaScript, GitHub Actions,
and C++ compound requirements without changing their configured timeouts.
Bounded reference scans now stop before requesting another match and skip
all later extraction phases on the same dense line once capacity is reached.
Symbol and dependency extractors now stream their remaining multi-match
patterns across scientific/native, Pascal/Ada, SQL, Python, Swift, GraphQL,
markup/XAML, shell, Ruby, Perl, Elixir, CSS, HDL, C++, and manifest parsing.
Count-only scans no longer retain a `MatchCollection`, while preserving the
previous all-or-nothing result if matching times out.
Dense XAML supplemental symbol scans now stop as soon as the shared
structured-data symbol budget needs its diagnostic marker, instead of
building an unbounded temporary list and trimming it after all phases.
Symbol container assignment now reuses one path buffer across members instead
of allocating a stack snapshot and a second list for every nested symbol.
Prolog goal scans now allocate per-line call lists only when a known goal is
emitted and update directive metadata in place instead of copying the list.
JSON symbol and reference extraction now build UTF-8 line-offset arrays at
exact capacity instead of retaining a growing list alongside its final copy.

## 日本語

- **bounded regex enumeration を demand-driven にしました** — 結果上限に達して
停止する extractor は、source line に残るすべての regex match を先に実体化せず、
right-to-left instance の既定の逆順も維持します。Fortran、Visual Basic、F#、
Pascal、Objective-C、Haskell、Elixir、Smalltalk、Lua、Dart、Razor、JSON、
JavaScript、GitHub Actions、C++ compound requirement の secondary reference
scanner も設定済み timeout を変えず同じ逐次経路を使います。bounded reference
scan は上限到達後に次の match を要求せず、同じ dense line の後続 extraction phase
も省略します。symbol / dependency extractor も scientific / native、Pascal / Ada、
SQL、Python、Swift、GraphQL、markup / XAML、shell、Ruby、Perl、Elixir、CSS、HDL、
C++、manifest parsing の残存 multi-match pattern を逐次走査します。count のみの scan
は `MatchCollection` を保持せず、matching timeout 時は従来どおり all-or-nothing の
結果を返します。dense XAML の supplemental symbol scan は、共有 structured-data
symbol budget の diagnostic marker が必要になった時点で停止し、全 phase の後まで
無制限の一時 list を構築してから trim しません。symbol の container assignment も
member ごとに stack snapshot と2つ目の list を割り当てず、1つの path buffer を再利用します。
Prolog goal scan も既知 goal を出力するときだけ per-line call list を割り当て、directive
metadata は list を copy せず in-place で更新します。JSON symbol / reference extraction
も UTF-8 line-offset array を exact capacity で構築し、成長中の list と最終 copy を同時に
保持しません。
30 changes: 30 additions & 0 deletions changelog.d/unreleased/+stream-core-reference-matches.fixed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
---
category: fixed
affected:
- src/CodeIndex/Indexer/References/Languages/CSharpReferenceExtractor.cs
- src/CodeIndex/Indexer/References/Languages/CSharpReferenceExtractor.Support.cs
- src/CodeIndex/Indexer/References/ReferenceExtractor.CoreCallReferences.cs
- src/CodeIndex/Indexer/References/ReferenceExtractor.CoreLanguageLines.cs
- src/CodeIndex/Indexer/References/ReferenceExtractor.CoreSpecializedLines.cs
- src/CodeIndex/Indexer/References/ReferenceExtractor.CoreTypeReferences.cs
- src/CodeIndex/Indexer/References/ReferenceExtractor.CSharpPatterns.cs
- src/CodeIndex/Indexer/References/ReferenceExtractor.CSharpTypeNames.cs
- src/CodeIndex/Indexer/References/ReferenceExtractor.PatternTypeReferences.cs
- src/CodeIndex/Indexer/References/ReferenceExtractor.PrimaryConstructors.cs
- src/CodeIndex/Indexer/References/ReferenceExtractor.ReferenceRecords.cs
- src/CodeIndex/Indexer/References/ReferenceExtractor.Solidity.cs
- DEVELOPER_GUIDE.md
- TESTING_GUIDE.md
---

## English

- **Streamed core reference matches** — Shared calls, C# attributes, types,
patterns and locals, JSX elements, JVM documentation links, and Solidity
scanners now consume single-pass match groups on demand.

## 日本語

- **core reference match を逐次走査にしました** — 共有 call、C# attribute / type /
pattern / local、JSX element、JVM documentation link、Solidity scanner は
single-pass の match group を demand-driven に消費します。
26 changes: 26 additions & 0 deletions changelog.d/unreleased/+stream-dynamic-reference-matches.fixed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
---
category: fixed
affected:
- src/CodeIndex/Indexer/References/Languages/PhpReferenceExtractor.cs
- src/CodeIndex/Indexer/References/Languages/PhpReferenceExtractor.LanguageTypes.cs
- src/CodeIndex/Indexer/References/Languages/PhpReferenceExtractor.Members.cs
- src/CodeIndex/Indexer/References/Languages/RubyReferenceExtractor.cs
- src/CodeIndex/Indexer/References/Languages/RReferenceExtractor.cs
- src/CodeIndex/Indexer/References/Languages/RReferenceExtractor.Members.cs
- src/CodeIndex/Indexer/References/Languages/RReferenceExtractor.CallsAndResources.cs
- src/CodeIndex/Indexer/References/Languages/PerlReferenceExtractor.cs
- DEVELOPER_GUIDE.md
- TESTING_GUIDE.md
---

## English

- **Streamed dynamic-language reference matches** — PHP, Ruby, R, and Perl
scanners now enumerate dense attribute, type, DSL, namespace/member/resource,
and arrow-call matches only while the reference budget has capacity.

## 日本語

- **dynamic-language の reference match を逐次走査にしました** — PHP、Ruby、R、
Perl scanner は dense な attribute、type、DSL、namespace / member / resource、
arrow-call match を reference budget に空きがある間だけ列挙します。
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
---
category: fixed
affected:
- src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Clojure.cs
- src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Erlang.cs
- src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Ocaml.cs
- src/CodeIndex/Indexer/References/Languages/FunctionalLanguageReferenceExtractor.Raku.cs
- src/CodeIndex/Indexer/References/Languages/ElixirReferenceExtractor.cs
- DEVELOPER_GUIDE.md
- TESTING_GUIDE.md
---

## English

- **Streamed functional-language reference matches** — Clojure, Elixir,
Erlang, OCAML, and Raku now enumerate regex results on demand and stop dense
lines as soon as the per-file reference budget is full.

## 日本語

- **functional-language の reference match を逐次走査にしました** — Clojure、
Elixir、Erlang、OCAML、Raku は regex result を demand-driven に列挙し、per-file
reference budget が満杯になると dense line の走査を停止します。
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
---
category: fixed
affected:
- src/CodeIndex/Indexer/References/Languages/BuildAutomationReferenceExtractor.cs
- src/CodeIndex/Indexer/References/Languages/CssReferenceExtractor.cs
- src/CodeIndex/Indexer/References/Languages/DockerfileReferenceExtractor.cs
- src/CodeIndex/Indexer/References/Languages/HdlReferenceExtractor.cs
- src/CodeIndex/Indexer/References/Languages/HdlReferenceExtractor.Scopes.cs
- src/CodeIndex/Indexer/References/Languages/MarkupSchemaReferenceExtractor.cs
- src/CodeIndex/Indexer/References/Languages/PowerShellReferenceExtractor.cs
- src/CodeIndex/Indexer/References/Languages/ShellReferenceExtractor.cs
- src/CodeIndex/Indexer/References/Languages/XamlReferenceExtractor.cs
- DEVELOPER_GUIDE.md
- TESTING_GUIDE.md
---

## English

- **Streamed infrastructure and markup reference matches** — CSS, XAML,
HTML/GraphQL/Markdown, HDL, MSBuild, Dockerfile, shell, and PowerShell
scanners now consume dense match groups on demand.

## 日本語

- **infrastructure / markup の reference match を逐次走査にしました** — CSS、
XAML、HTML / GraphQL / Markdown、HDL、MSBuild、Dockerfile、shell、PowerShell
scanner は dense な match group を demand-driven に消費します。
Loading
Loading