Span based scrubbers#1804
Merged
Merged
Conversation
Replaces the StringBuilder based scrubbing pipeline with a span based engine that materializes the document once, tracks it as chunks, and quarantines replacements so other scrubbers never reprocess them. New public API: Scrubber.Replace / Window / Match / RemoveLinesContaining / RemoveLines / ReplaceLines / RemoveEmptyLines, registered via AddScrubber(Scrubber) at global, instance, and extension mapped levels. - Ordering is engine determined: line drops, line transforms, then inline scrubbers (unknown max length first, then longest max first); directory replacements stay pinned last so scrubbers always see raw paths - Known min lengths skip short values; unchanged input is returned zero copy (the property value fast path now actually fires: the TempFile/TempDirectory module init scrubbers had made the old GlobalScrubbers.Count check permanently false) - All built-in scrub methods reroute to the engine; ScrubberLocation overloads are obsolete and ignored. Legacy AddScrubber(Action<StringBuilder>) is unchanged and runs after the engine, only when registered - Deletes ~900 lines of chunk carryover code (GuidScrubber, DateScrubber scan loops, LinesScrubber, UserMachineScrubber PerformReplacements, DirectoryReplacements_StringBuilder) - Date format validation probes 2000-01-01 instead of DateTime.MaxValue, which is out of range for the UmAlQura calendar Perf (31KB doc, Ryzen 5900X): guid scan 6.6x faster with 51x less allocation; ISO DateTime scan 6.8x / 100x; DateTimeOffset 11x; guids+dates+line drops composed in one pass 5.4x / 3.5x. New ApplyScrubbersTests project covers engine semantics; all suites pass with zero snapshot churn.
string.StartsWith(string) and string.EndsWith(string) default to StringComparison.CurrentCulture, so these were running culture sensitive ICU collation to match machine readable tokens: date format specifiers, assembly name prefixes, editorconfig section headers, and the section markers of the Verify exception message format. Ordinal is both faster and stricter here. Culture sensitive comparison can treat some characters as ignorable, so a prefix could match while consuming a different number of characters than the prefix contains. Parser.TrimStart relied on that count, slicing with next[prefix.Length..] after a culture sensitive StartsWith. Only string receivers are affected. The StartsWith calls on ReadOnlySpan<char> in ScrubStackTrace and InnerVerifyChecks already use ordinal sequence comparison, and IndexOf(char) in Guards is ordinal, so those are left alone.
The scrub engine section named the Scrubber factory methods but had no example, so the new AddScrubber(Scrubber) overloads were the only scrubber APIs without a compiled snippet. Add one covering Scrubber .Replace and Scrubber.Window. Also document extension scoped scrubbers, which had no coverage at all. That gap predates the engine: the extension mapped AddScrubber(Action<StringBuilder>) overloads were already undocumented. The snippets live in the existing compile checked List method, so the examples cannot drift from the API without breaking the build. guids.md, members-throw.md, named-tuples.md, obsolete-members.md and serializer-settings.md are regenerated only because the added lines shift the snippet source line links in SerializationTests.cs.
The AddScrubber snippet region already existed but no doc referenced it, so the legacy overloads were described in the intro without an example. Add a Legacy scrubbers section that adopts the region, alongside the scrub engine section.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This change replaces the internal text-scrubbing pipeline with a new span based scrub engine and
adds new
ScrubReplace/ScrubWindow/ScrubMatchmethods for defining customscrubbers. The engine materializes each document once, tracks it as chunks, and quarantines
replacements so scrubbers no longer re-process each other's output. The result is large CPU and
allocation reductions for the built-in scrubbers, plus a first-class way to define custom scrubbers.
The full behavior is documented in scrubbers.md.
Why
The old pipeline was
StringBuilderbased. Every registered scrubber re-round-tripped the wholedocument (
ToString()+ rebuild, or chunk-walking with cross-chunk carryover buffers), so cost scaledwith
scrubbers x document sizeand allocated heavily even when nothing matched. The new engine does asingle pass with vectorized candidate scanning and returns the original string unchanged (zero copy)
when no scrubber fired.
Performance
Measured on a 31 KB document (p99 of a scan of 33,707 real
.verified.files), AMD Ryzen 9 5900X,.NET 10,
MemoryDiagnoser. "Old" runs the pre-engine implementations over identical input.ScrubLinesContaining, 508 linesThe one scenario that regresses is predicate line removal (
ScrubLines(Func<string, bool>)) on small andmedium documents (~0.5 us at p50, ~10 us at 1,000 lines) — the cost of the composable pipeline. It wins
at scale (5x at 10,000 lines) and is avoidable via the new span predicate overload or
ScrubLinesContaining.Changes
API (source compatible, with obsoletions)
No public API was removed and no signatures changed incompatibly. However, the
ScrubberLocationparameter is now meaningless for the built-in scrub methods (ordering is engine determined), so those
overloads are marked
[Obsolete]:ScrubLinesContaining,ScrubLines,ScrubLinesWithReplace,ScrubEmptyLines,ScrubInlineGuids,ScrubInlineDates,ScrubInlineDateTimes,ScrubInlineDateTimeOffsets,ScrubMachineName,ScrubUserName(at global, instance, extension mapped, and fluent levels).Existing calls still compile, but a call that passes a
ScrubberLocationproduces an obsolete warning.Projects using
TreatWarningsAsErrorswill need to drop theScrubberLocationargument. TheScrubberLocationon the low-levelAddScrubber(Action<StringBuilder>, ...)overloads is unchangedand still honored.
Behavior (snapshot output)
Even source-identical code can now produce different verified output in these cases. Where it applies,
re-accept the affected
.verified.files.ScrubberLocation.First/.Laston abuilt-in scrub method is ignored. Registration level (global vs instance) no longer grants broad
priority; inline scrubbers order by match length.
AddScrubber(Action<StringBuilder>)scrubbers, regardlessof registration order. Custom
AddScrubberdelegates run after the engine and still see (and canmodify) its output.
ScrubLinesWithReplacenow compose in registration order (FIFO). Previously the defaultScrubberLocation.Firstcomposed them in reverse.another. Legacy
AddScrubberdelegates are unaffected and can still re-scrub.!char.IsLetterOrDigit. Inline guid scrubbing previously usedchar.IsLetter || char.IsNumber; a guid adjacent to an exotic numeric character (for example asuperscript digit) is now scrubbed where it previously was not.
\r\nand lone\rbecome\n) before scrubbers run. A legacyAddScrubberdelegate that matches a literal"\r\n"will no longer match.replacement boundary. Only relevant when another scrubber's replacement sits immediately after a
scrubbed path.
Scrubbers that cannot be expressed on the engine (positional buffer edits, full-document reformatters,
multi-line regex) should stay on the
AddScrubber(Action<StringBuilder>)overloads, which are unchanged.Migration
ScrubberLocationargument from built-in scrubcalls, for example
ScrubMachineName(ScrubberLocation.Last)becomesScrubMachineName()..verified.files that changed. Expected only in theconfigurations listed under "Behavior" above; a suite that does not rely on scrubber ordering or
sugar-vs-legacy interaction should see no changes.
allocation, for example
ScrubLines((ReadOnlySpan<char> line) => ...).New public API
Custom engine scrubbers follow the existing
Scrub[Thing]convention and are available at every level(global, instance, extension mapped, fluent). Three new methods expose the engine's inline matching
primitives:
Extension scoped variants take the extension as the first argument, for example
settings.ScrubReplace("json", "find", "replacement").ScrubLinesandScrubLinesWithReplacealso gained span-delegate overloads (LineMatch/LineReplace); select them with an explicitly typed lambda parameter ((ReadOnlySpan<char> line) => ...). Untypedlambdas continue to bind the existing
string-based overloads. ALineReplacereturnsLineResult.Keep/LineResult.Remove/LineResult.Replace(text).The
Scrubbertype that backs the engine is internal; all registration goes through theScrub*methods.
See scrubbers.md for full semantics.