Speed up indexing for large codebases

[Widthdom]

Summary

• stream bounded regex and reference-match enumeration so capped scans stop before requesting an unused next match
• replace line arrays, temporary collections, closures, and repeated string operations with spans, exact-capacity buffers, reusable path storage, and early exits
• apply the improvements consistently across core indexing, symbol extraction, dependency extraction, and the supported language families
• add allocation, cap-boundary, symbol-extraction, and regex-enumeration coverage together with changelog and testing documentation

Root cause

Large repositories amplified several small costs in hot indexing loops: materialized regex match collections, line arrays, temporary lists and predicates, repeated trimming and suffix checks, and scans that advanced once more after reaching their output budget. Together these produced substantial cumulative managed allocation and unnecessary work during symbol and reference extraction.

Impact

Indexing work is now streamed and bounded earlier, with fewer transient allocations. A representative full-repository measurement improved total elapsed time from 54,918 ms to 54,027 ms while indexing 24 additional branch files, and reduced cumulative managed allocation from 13,207,092,368 bytes to 13,058,802,272 bytes. The bounded-match wrapper is a struct-based enumerable, preserving the allocation-free below-cap path while checking capacity before the underlying MoveNext() call.

Validation

• Debug build: 0 warnings, 0 errors
• Release full test suite:
  • .NET 8: 10,472 passed, 0 failed, 7 skipped
  • .NET 9: 9,999 passed, 0 failed, 419 skipped
• focused reference-extractor suite:
  • .NET 8: 1,262 passed
  • .NET 9: 1,257 passed, 5 skipped
• final local index update: 1,216 files, 46,124 symbols, 354,582 references, 0 warnings, 0 errors, 0 cap hits
• changelog validation, formatting verification, and git diff --check passed
• independent adversarial review found no blocking or actionable issues