feat(atomsplit): run deepseek's pre-tokenizer with no regex backend - #2265

Draft

SBrandeis wants to merge 6 commits into

feat/train_encode_splitfrom

Commits on Jul 29, 2026

new literal module in AtomSplit
SBrandeis
committed
059bcb8
use Literal in Replace normalizer and Split pretok
SBrandeis
committed
f01e165
box the Literal finder
SBrandeis
committed
1a38391
keep the &str / &String patterns, backed by Literal
SBrandeis
committed
179799c
feat(atomsplit): run deepseek's pre-tokenizer with no regex backend

SBrandeis
and
claude
committed
f4ac843
bench: measure the pipeline in the configuration it ships

SBrandeis
and
claude
committed
780b71d