Skip to content

Use System.IO.Enumeration for faster recursive path traversal#112

Merged
kthompson merged 1 commit into
developfrom
kthompson-io-enumeration-path-traversal
Jun 27, 2026
Merged

Use System.IO.Enumeration for faster recursive path traversal#112
kthompson merged 1 commit into
developfrom
kthompson-io-enumeration-path-traversal

Conversation

@kthompson

@kthompson kthompson commented Jun 27, 2026

Copy link
Copy Markdown
Owner

Why

The old traverser used a manual BFS loop that called GetFiles() / GetDirectories() per directory, allocating DirectoryInfo[] arrays and FileSystemInfo objects for every entry regardless of whether it matched the glob pattern. For sparse patterns like **/*.csproj, this resulted in large allocations (Gen2 GC pressure) and slow traversal.

This PR rewires path traversal to use a single FileSystemEnumerable<FileSystemInfo> with RecurseSubdirectories = true, pushing glob matching directly into the enumeration predicates.

Approach

Two new types do the work:

  • PathMatcher -- span-based IsFullMatch and CanRecurse methods that mirror GlobEvaluator's ** backtracking semantics without allocating a string array for path segments.
  • RecursiveGlobEnumerable -- wraps a single FileSystemEnumerable with:
    • ShouldRecursePredicate calling PathMatcher.CanRecurse to prune entire subtrees that cannot possibly match the pattern
    • ShouldIncludePredicate calling PathMatcher.IsFullMatch for the full match check
    • Transform (ToFileSystemInfo) only runs on matched entries

Each path is visited exactly once by the OS, so no HashSet dedup is needed. Entries that don't match are never materialized into FileSystemInfo objects.

The one behavioral subtlety: a bare ** pattern also matches the search root itself (zero directories deep). Since FileSystemEnumerable never visits the root, RecursiveGlobEnumerable emits it explicitly when the pattern is all-wildcard.

Removed dead code

With the new traverser in place, the old per-directory machinery is no longer needed:

  • Deleted PathTraverserEnumerable.cs (manual BFS)
  • TraverseOptions reduced to a plain flags carrier (no GetFiles/GetDirectories, no _dirCache)
  • Deleted MockTraverseOptions, MockFileSystemNode, and their unit tests
  • Removed System.IO.Abstractions package references from Glob.Tests

Benchmark results (net8.0, Apple M3)

Comparing against origin/develop with identical benchmark definitions ([MemoryDiagnoser] added to baseline for fair comparison):

Benchmark Baseline This PR Time Alloc
ParseGlob 1,474 ns 1,461 ns -1% --
ParseAndCompileGlob 1,495 ns 1,513 ns +1% --
MatchForUncompiledGlob 1,759 ns 1,772 ns +1% --
MatchForCompiledGlob 199 ns 189 ns -5% --
MatchForUncompiledGlobDirectoryWildcard 1,630 ns 1,635 ns +0% --
MatchForCompiledGlobDirectoryWildcard 200 ns 190 ns -5% --
BenchmarkParseToTree 1,525 ns 1,561 ns +2% --
PathTraversal (**/*.cs) 29,635 ns 30,148 ns +2% -18%
PathTraversalSparseMatch (**/*.csproj) 1,622,130 ns 714,784 ns -56% -70%
PathTraversalDirectories (**/bin) 689,596 ns 535,730 ns -22% -46%

Parse/match benchmarks are within noise and allocations are unchanged. Gen2 GC pressure disappears entirely for sparse traversal patterns.

Test coverage

All 179 tests pass (183 minus the 4 MockTraverseOptions unit tests that tested the now-deleted code). Real-filesystem coverage in PathTraverserTests continues to exercise all traversal contracts: dedup, symlink handling, re-enumeration, case-insensitivity, emitFiles/emitDirectories flags, rooted patterns.

Closes #33

@kthompson
kthompson marked this pull request as ready for review June 27, 2026 19:03
Replaces the manual BFS traverser with a single FileSystemEnumerable
using RecurseSubdirectories. Glob matching is pushed into the enumeration:

- ShouldRecursePredicate (PathMatcher.CanRecurse) prunes subtrees that
  cannot match the pattern before any IO is done on them
- ShouldIncludePredicate (PathMatcher.IsFullMatch) performs the full glob
  match; only matched entries are materialized into FileSystemInfo objects
- Each path is visited exactly once -- no HashSet dedup needed

PathMatcher uses span-based matching (no string[] allocation per path)
mirroring GlobEvaluator's ** backtracking semantics. A bare ** pattern
still emits the search root for behavioral parity.

Dead code removed: PathTraverserEnumerable (manual BFS), GetFiles/
GetDirectories + _dirCache in TraverseOptions (now a plain flags carrier),
MockTraverseOptions/MockFileSystemNode and their System.IO.Abstractions
package dependencies.

Benchmarks (net8.0, Apple M3) vs origin/develop:
  PathTraversalSparseMatch  1,622,130 ns -> 714,784 ns  (-56%, -70% alloc)
  PathTraversalDirectories    689,596 ns -> 535,730 ns  (-22%, -46% alloc)
  PathTraversal                29,635 ns ->  30,148 ns  (+2%,  -18% alloc)

Closes #33

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@kthompson
kthompson force-pushed the kthompson-io-enumeration-path-traversal branch from e533151 to 053f726 Compare June 27, 2026 19:17
@kthompson
kthompson enabled auto-merge (rebase) June 27, 2026 19:18
@kthompson
kthompson merged commit 719a859 into develop Jun 27, 2026
3 checks passed
@kthompson
kthompson deleted the kthompson-io-enumeration-path-traversal branch June 27, 2026 19:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Consider use of System.IO.Enumeration

1 participant