feat(atomsplit): run deepseek's pre-tokenizer with no regex backend - #2265
Draft
SBrandeis wants to merge 6 commits into
Draft
feat(atomsplit): run deepseek's pre-tokenizer with no regex backend#2265SBrandeis wants to merge 6 commits into
SBrandeis wants to merge 6 commits into
Conversation
`memmem::Finder` carries a few hundred bytes of prefilter state on x86_64 (much less on aarch64, which is why this only showed up in CI): a `Literal` stored inline blew `NormalizerWrapper` and `DecoderWrapper` up to 352 bytes through `Replace`, which `clippy::large_enum_variant` rejects.
The Python bindings search with a `&String` pattern — `NormalizedString.replace` and `.split` take a plain `str` — so dropping these impls broke every job that builds the bindings. Reinstating them through `Literal` keeps the public API and still drops the regex engine from the literal path: they used to escape the string and compile a regex on every call. An empty pattern now covers the input by byte length rather than character count, like every other impl (the old count sliced mid-character).
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
deepseek's `tokenizer.json` failed to LOAD without the `fancy-regex` feature.
Its pre-tokenizer is a `Sequence` of three `Split`s, and `atomsplit` only knew
the three as a *chain* (`fsm_deepseek`, the fused pass). `Split::new` accepts a
missing backend only for a pattern it recognizes on its own, and no single
deepseek pattern was recognized — so deserialization errored out before the
`Sequence` ever got the chance to spot the chain.
Each split now has its own byte-exact FSM, so each `Split` is recognized alone:
fsm_deepseek_num `\p{N}{1,3}`
fsm_deepseek_cjk `[一-龥-ゟ゠-ヿ]+`
fsm_deepseek_big the big regex
`fsm_deepseek` keeps fusing all three when the whole chain is present; it and
`fsm_deepseek_big` share one body under two compile-time flags saying which
earlier splits are folded in, so the grammar lives in one place. Measured
performance-neutral on `fsm_deepseek` (median +0.02% over 10 corpora, separate
binaries, alternating runs — the microbench's layout noise is several times
that).
Also:
- `DEEPSEEK_BIG` now carries literal CR/LF, matching the string deepseek ships,
so `tk-encode` no longer keeps a second copy of the three patterns just to
compare against them; `utils::is_deepseek` is gone (recognition is per-split).
- `GptFsm::split_into` replaces the duplicated fsm `match` in the legacy and
pipeline paths.
- `canonicalized_for_pipeline` now rewrites `(invert, Removed)` →
`(!invert, Isolated)` only for patterns that match the whole input
(`GptFsm::covers_input`). deepseek's leave gaps, where the two forms differ.
- The three deepseek `Sequence` tests pointed at a fixture name that no longer
exists, so they had been silently skipping; fixed, and un-gated from
`fancy-regex` since the splits now build without it (same for the dsv4
pipeline bench). With the feature on they still compare against the real
regex engine; with it off they pin fused == chained.
Tests: per-split onig parity over both parity corpora + 9 Wikipedia languages,
chain-equals-fused over the same, and a `tokenizer.json` load+encode test whose
ids must match the legacy path in either build.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The CI benchmark built `fixture_bench` with `bench-baseline`, which enabled
`fancy-regex`. So the throughput, memory and model-coverage numbers described a
build nobody ships, while the binary size next to them was measured on a
different binary (`binsize_pipeline`, built without it). The backend is not a
rounding error: on a minimal encode program it is +1.47 MB stripped, +71%.
`fancy-regex` was in that feature only to supply one of the *reference regex
engines* the report times our split against. Those move to their own binary:
bench-baseline = the released crate, to compare against -> fixture_bench
bench-engines = onig + PCRE2 + fancy-regex + logos -> pretok_engines
`fixture_bench` now builds with no regex backend, and the shared work list —
corpus loading, the model manifest, the timing helpers, `--shard` — lives in
`examples/bench_common/` so a number from one binary is comparable to the other.
The report is unchanged: each shard runs both binaries and the `report` job folds
the engine numbers back onto the row they describe as `pretok_vs_regex`, which is
where the renderer already looks. Verified end to end locally — merge attaches
198 rows (9 models x 22 fixtures) and the renderer emits the same tables.
All 10 manifest models load with no backend, which is what makes this possible:
deepseek needed per-split FSMs (previous commit) and the SentencePiece pair
(gemma-4, llama-2) needed literal patterns, whose `Split(" ")` / `Replace(" ",
"▁")` are not regexes at all. `tests/no_regex_backend.rs` pins that, so adding a
model that needs an engine fails a test instead of quietly turning into an error
card in the report.
Ran the benchmark this way over all 22 fixtures: 9/10 models benched with
ids_match true against the released crate; t5-base is the pre-existing Metaspace
pipeline gap, unrelated to the backend.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
SBrandeis
changed the base branch from
feat/train_encode_split
to
feat/literal-patterns
July 30, 2026 08:53
Base automatically changed from
feat/literal-patterns
to
feat/train_encode_split
July 31, 2026 14:15
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.