Skip to content

Span based scrubbers#1804

Merged
SimonCropp merged 41 commits into
mainfrom
span-based-scrubbers
Jul 19, 2026
Merged

Span based scrubbers#1804
SimonCropp merged 41 commits into
mainfrom
span-based-scrubbers

Conversation

@SimonCropp

@SimonCropp SimonCropp commented Jul 18, 2026

Copy link
Copy Markdown
Member

This change replaces the internal text-scrubbing pipeline with a new span based scrub engine and
adds new ScrubReplace / ScrubWindow / ScrubMatch methods for defining custom
scrubbers. The engine materializes each document once, tracks it as chunks, and quarantines
replacements so scrubbers no longer re-process each other's output. The result is large CPU and
allocation reductions for the built-in scrubbers, plus a first-class way to define custom scrubbers.

The full behavior is documented in scrubbers.md.

Why

The old pipeline was StringBuilder based. Every registered scrubber re-round-tripped the whole
document (ToString() + rebuild, or chunk-walking with cross-chunk carryover buffers), so cost scaled
with scrubbers x document size and allocated heavily even when nothing matched. The new engine does a
single pass with vectorized candidate scanning and returns the original string unchanged (zero copy)
when no scrubber fired.

Performance

Measured on a 31 KB document (p99 of a scan of 33,707 real .verified. files), AMD Ryzen 9 5900X,
.NET 10, MemoryDiagnoser. "Old" runs the pre-engine implementations over identical input.

Scenario Old New Speedup Allocation
Inline guid scan, no match 105.5 us / 61.8 KB 6.6 us / 1.2 KB 16x 51x less
Inline DateTime (ISO) scan, no match 709.9 us / 122.4 KB 72.7 us / 1.2 KB 9.8x 100x less
Inline DateTimeOffset, typical 69.7 us / 15.5 KB 4.1 us / 10.8 KB 17x
ScrubLinesContaining, 508 lines 16.2 us / 186 KB 10.3 us / 59 KB 1.6x 3.2x less
Whole pipeline (guids + dates + line drops, one pass) 884.0 us / 359 KB 145.1 us / 103 KB 6.1x 3.5x less
Serialization string probe (per value, all misses) 144 ns 42 ns 3.4x

The one scenario that regresses is predicate line removal (ScrubLines(Func<string, bool>)) on small and
medium documents (~0.5 us at p50, ~10 us at 1,000 lines) — the cost of the composable pipeline. It wins
at scale (5x at 10,000 lines) and is avoidable via the new span predicate overload or
ScrubLinesContaining.

Changes

API (source compatible, with obsoletions)

No public API was removed and no signatures changed incompatibly. However, the ScrubberLocation
parameter is now meaningless for the built-in scrub methods (ordering is engine determined), so those
overloads are marked [Obsolete]:

ScrubLinesContaining, ScrubLines, ScrubLinesWithReplace, ScrubEmptyLines, ScrubInlineGuids,
ScrubInlineDates, ScrubInlineDateTimes, ScrubInlineDateTimeOffsets, ScrubMachineName,
ScrubUserName (at global, instance, extension mapped, and fluent levels).

Existing calls still compile, but a call that passes a ScrubberLocation produces an obsolete warning.
Projects using TreatWarningsAsErrors will need to drop the ScrubberLocation argument. The
ScrubberLocation on the low-level AddScrubber(Action<StringBuilder>, ...) overloads is unchanged
and still honored.

Behavior (snapshot output)

Even source-identical code can now produce different verified output in these cases. Where it applies,
re-accept the affected .verified. files.

  1. Order among built-in scrubbers is engine determined. ScrubberLocation.First / .Last on a
    built-in scrub method is ignored. Registration level (global vs instance) no longer grants broad
    priority; inline scrubbers order by match length.
  2. Built-in scrubbers run before legacy AddScrubber(Action<StringBuilder>) scrubbers, regardless
    of registration order. Custom AddScrubber delegates run after the engine and still see (and can
    modify) its output.
  3. Multiple ScrubLinesWithReplace now compose in registration order (FIFO). Previously the default
    ScrubberLocation.First composed them in reverse.
  4. Quarantine. Text produced by one engine scrubber replacement is no longer re-examined by
    another. Legacy AddScrubber delegates are unaffected and can still re-scrub.
  5. Word boundary rule unified to !char.IsLetterOrDigit. Inline guid scrubbing previously used
    char.IsLetter || char.IsNumber; a guid adjacent to an exotic numeric character (for example a
    superscript digit) is now scrubbed where it previously was not.
  6. Text is newline-normalized (\r\n and lone \r become \n) before scrubbers run. A legacy
    AddScrubber delegate that matches a literal "\r\n" will no longer match.
  7. Directory/path replacement greedy trailing-separator absorption does not cross a quarantined
    replacement boundary.
    Only relevant when another scrubber's replacement sits immediately after a
    scrubbed path.

Scrubbers that cannot be expressed on the engine (positional buffer edits, full-document reformatters,
multi-line regex) should stay on the AddScrubber(Action<StringBuilder>) overloads, which are unchanged.

Migration

  1. If the build fails with obsolete warnings, remove the ScrubberLocation argument from built-in scrub
    calls, for example ScrubMachineName(ScrubberLocation.Last) becomes ScrubMachineName().
  2. Run the test suite and re-accept any .verified. files that changed. Expected only in the
    configurations listed under "Behavior" above; a suite that does not rely on scrubber ordering or
    sugar-vs-legacy interaction should see no changes.
  3. Optional: for hot custom line predicates, switch to the new span overloads to avoid a per-line string
    allocation, for example ScrubLines((ReadOnlySpan<char> line) => ...).

New public API

Custom engine scrubbers follow the existing Scrub[Thing] convention and are available at every level
(global, instance, extension mapped, fluent). Three new methods expose the engine's inline matching
primitives:

// Fixed-string replacement (optional ordinal comparison / word boundary; multi-pair overload available)
settings.ScrubReplace("find", "replacement");

// Sliding window matcher (used internally by the guid and date scrubbers)
settings.ScrubWindow(minLength: 26, maxLength: 26, (window, counter, context) => ...);

// General search matcher (locates the next match within a segment)
settings.ScrubMatch((segment, counter, context, out index, out length, out replacement) => ...);

Extension scoped variants take the extension as the first argument, for example
settings.ScrubReplace("json", "find", "replacement").

ScrubLines and ScrubLinesWithReplace also gained span-delegate overloads (LineMatch /
LineReplace); select them with an explicitly typed lambda parameter ((ReadOnlySpan<char> line) => ...). Untyped
lambdas continue to bind the existing string-based overloads. A LineReplace returns
LineResult.Keep / LineResult.Remove / LineResult.Replace(text).

The Scrubber type that backs the engine is internal; all registration goes through the Scrub*
methods.

See scrubbers.md for full semantics.

SimonCropp and others added 30 commits July 11, 2026 13:56
Replaces the StringBuilder based scrubbing pipeline with a span based
engine that materializes the document once, tracks it as chunks, and
quarantines replacements so other scrubbers never reprocess them.

New public API: Scrubber.Replace / Window / Match / RemoveLinesContaining /
RemoveLines / ReplaceLines / RemoveEmptyLines, registered via
AddScrubber(Scrubber) at global, instance, and extension mapped levels.

- Ordering is engine determined: line drops, line transforms, then inline
  scrubbers (unknown max length first, then longest max first); directory
  replacements stay pinned last so scrubbers always see raw paths
- Known min lengths skip short values; unchanged input is returned
  zero copy (the property value fast path now actually fires: the
  TempFile/TempDirectory module init scrubbers had made the old
  GlobalScrubbers.Count check permanently false)
- All built-in scrub methods reroute to the engine; ScrubberLocation
  overloads are obsolete and ignored. Legacy AddScrubber(Action<StringBuilder>)
  is unchanged and runs after the engine, only when registered
- Deletes ~900 lines of chunk carryover code (GuidScrubber,
  DateScrubber scan loops, LinesScrubber, UserMachineScrubber
  PerformReplacements, DirectoryReplacements_StringBuilder)
- Date format validation probes 2000-01-01 instead of DateTime.MaxValue,
  which is out of range for the UmAlQura calendar

Perf (31KB doc, Ryzen 5900X): guid scan 6.6x faster with 51x less
allocation; ISO DateTime scan 6.8x / 100x; DateTimeOffset 11x;
guids+dates+line drops composed in one pass 5.4x / 3.5x.

New ApplyScrubbersTests project covers engine semantics; all suites pass
with zero snapshot churn.
string.StartsWith(string) and string.EndsWith(string) default to
StringComparison.CurrentCulture, so these were running culture sensitive
ICU collation to match machine readable tokens: date format specifiers,
assembly name prefixes, editorconfig section headers, and the section
markers of the Verify exception message format.

Ordinal is both faster and stricter here. Culture sensitive comparison
can treat some characters as ignorable, so a prefix could match while
consuming a different number of characters than the prefix contains.
Parser.TrimStart relied on that count, slicing with next[prefix.Length..]
after a culture sensitive StartsWith.

Only string receivers are affected. The StartsWith calls on
ReadOnlySpan<char> in ScrubStackTrace and InnerVerifyChecks already use
ordinal sequence comparison, and IndexOf(char) in Guards is ordinal, so
those are left alone.
The scrub engine section named the Scrubber factory methods but had no
example, so the new AddScrubber(Scrubber) overloads were the only
scrubber APIs without a compiled snippet. Add one covering Scrubber
.Replace and Scrubber.Window.

Also document extension scoped scrubbers, which had no coverage at all.
That gap predates the engine: the extension mapped
AddScrubber(Action<StringBuilder>) overloads were already undocumented.

The snippets live in the existing compile checked List method, so the
examples cannot drift from the API without breaking the build.

guids.md, members-throw.md, named-tuples.md, obsolete-members.md and
serializer-settings.md are regenerated only because the added lines
shift the snippet source line links in SerializationTests.cs.
The AddScrubber snippet region already existed but no doc referenced it,
so the legacy overloads were described in the intro without an example.
Add a Legacy scrubbers section that adopts the region, alongside the
scrub engine section.
@SimonCropp SimonCropp added this to the 31.26.0 milestone Jul 19, 2026
@SimonCropp
SimonCropp merged commit 49aa184 into main Jul 19, 2026
4 of 6 checks passed
@SimonCropp
SimonCropp deleted the span-based-scrubbers branch July 19, 2026 12:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants