bitsplit: one crate, all model grammars, atomsplit deleted - #2317
Merged
Conversation
…ctness gate bitsplit had no tests at all -- lib.rs pointed at a `src/bin/verify.rs` that is not in the tree. Add `tests/parity.rs` first, so the rest of the cleanup has a gate: oniguruma as the oracle (deepseek composed as HF applies it, three Isolated splits), plus the block-phase sweep that is the only thing that actually exercises the cross-block carries and `starts[bi-1] |= patch`. A `#[should_panic]` negative control keeps the gate honest. All three existing grammars pass. Then move the classify cluster (classify, atom_tables, tables, the three SIMD kernels), `regexes` and `literal` into bitsplit, and put `Span` in its lib.rs. atomsplit is now a shim over bitsplit holding only the scalar FSMs; it goes away once every caller is rewired. Also resolves a bad merge already on this branch: tk-encode/src/models/bpe/model.rs carried two `fn tokenize_spans`, so tk-encode did not compile at all. Kept the fold_id/symbols mechanics (the ones matching BpeScratch and merge_word's arity) and restored the word-cache logic the other side had, mirroring tokenize_pipeline.
…imi), 4th plane, aux streams
Adds `o200k.rs`, const-generic over <AUX, CONTRACTION, DIGIT_CAP>. AUX doubles as the
variant selector because the two text-derived streams are mutually exclusive: the
o200k/tekken line needs `/` for rule 4's `[\r\n/]*`, kimi needs `\p{Han}` for its
leading arm and has a plain `[\r\n]*` tail.
Kernel changes:
- `Blk` gains `p3`. o200k needs 9 tag classes and code 7 is reserved -- both SIMD
kernels find continuation lanes by testing `lut[tag] == 7` -- so "other" cannot be
the leftover code. Const-gated off for the 3-plane grammars.
- `Blk.cjk` generalised to `Blk.aux` with a const AUX selector (none/cjk/slash/han).
- `digit_groups::<CAP>` lifted into lib.rs: the `\p{N}{1,CAP}` block was duplicated
verbatim in cl100k and deepseek, and hardcoding 3 is what blocked Qwen.
The case split is a scalar escape, not bit algebra: `[UC]*[LC]+|[UC]+[LC]*` has no
local form (`中Qz` is one token, `ʰABC` is two -- the difference is whether an L
appears LATER in the run). The bit half computes a cheap gate instead, so
all-lowercase and Capitalised text never pays for it.
o200k/tekken/kimi parity is #[ignore]d with the reason: \p{M} is in BOTH the letter
classes and rule 4's class, and that interaction is not modelled yet. The other four
gates (gpt2, cl100k, deepseek, negative control) stay green.
The duplicate-tokenize_spans repair I wrote landed upstream as #2314 on feat/train_encode_split, which is NOT an ancestor of poc/target-encode-clean -- hence the stale base. Take the upstream text so the two cannot diverge; it also documents why the fold must be probed before the cache.
Drop the <AUX, CONTRACTION, DIGIT_CAP> const generics. Folding o200k, tekken and kimi into one parameterised grammar meant every fix was a three-way risk and nothing could be read on its own. Now: o200k.rs (llama4 / gpt-oss / minimax), tekken.rs, kimi.rs -- each unrolled against the shared primitives in lib.rs, each independently debuggable. The small scalar helpers (member, run_end, ws_tail, letter_match) move to lib.rs; they are genuinely shared and copying them per file would be worse. kimi is byte-exact. o200k/tekken still carry two cross-block bugs.
Three real bugs, all in the interaction between rule 4's `[\r\n/]*` tail and everything
around it. `/` is in BOTH the `+` body and the tail, which cl100k never has to deal with
(its tail is newlines, and a newline is not "other"):
- One tail run can collect SEVERAL markers (`!\n/\n`: the `\n` after `!` and the `\n`
after `/`). `nl_e - nl_m` spans only from the last one -- that is exactly what
`fill_to_last` exists for.
- A char the tail absorbed is not part of the `+` body, so an "other" after it opens a
fresh run (`<emoji>\r\n/#` = tail, then `#` starts again). Needs a `prev_absorbed` carry
for the block edge.
- ...but a char INSIDE the tail must not open one either, or `osf` picks up the `/` and
the `[^\r\n\p{L}\p{N}]?` prefix rule then eats the letter start after it (`#\r\n/aA`
came out as one token instead of `a` + `A`).
Plus: re-flag the last letter token whenever the escape trigger fires, since the trigger
can land in a later block than the token it belongs to.
Gate: oniguruma oracle, block-phase sweep, 4000 fuzz strings (10000 run locally, clean).
All 7 grammars green -- gpt2, cl100k, deepseek, o200k, tekken, kimi + negative control.
One file per regex, finishing the split. cl100k's digit cap becomes a plain argument rather than a const generic -- 3 = cl100k / Llama-3 / GLM-4.6, 1 = Qwen. cl100k was still running its own inline hardcoded-3 digit block; it now shares digit_groups with o200k, which is what made the cap knob real. 8 gates green: gpt2, cl100k, qwen, deepseek, o200k, tekken, kimi + negative control.
The regex FSMs are gone -- every recognized pattern now routes to the byte-exact bitstream grammar. `fast_builder()` went with them: there is no second engine left to route to, so the gate had nothing to gate. - `GptFsm` -> `Grammar`, one variant per distinct regex, with the Kimi key added. The cl100k digit cap now actually reaches the splitter (Qwen routes to `bitsplit_qwen`). - The class-run family and `CharDelimiterSplit` move to `bitsplit::classes`. - Workspace members drop atomsplit; bitmap_gen writes bitsplit/src/atom_tables.rs; the CI workflow is renamed. Whole workspace green: 8 bitsplit gates, 356 tk-encode unit tests, the bpe/decode/ pipeline oracles.
…rSplit Replaces memmem with the same shape as the grammars: a `u64` match bitmap per 64-byte block, walked with `trailing_zeros`. The first needle byte is compared in SIMD (the existing `mm64` fold on NEON, `movemask` on SSE); the remaining bytes only ever run on the survivors, so a 3-byte needle costs one vector pass plus a handful of scalar checks. This is what puts metaspace (llama2 / gemma) fully on bitsplit: `to_normalizer_and_split` already decomposes Metaspace at load time into a MetaspaceNormalizer + a `Split` on the `U+2581` delimiter, and that Split's search is now a bitstream. Measured on the metaspace needle over 8 MB (examples/litbench): scalar first byte 727 MB/s 0.64x memmem <- not good enough SIMD first byte 1433 MB/s 1.27x memmem memchr is no longer a bitsplit dependency.
The gate was "any trigger in this block flags every letter token in it", so a single `'s` in a paragraph dragged every word through the scalar pass. Narrow it with a reverse fill over the letter stream: a run needs the escape only if an interior upper (or the letter before a contraction apostrophe) sits at-or-after the run start. `last_lt` keeps tracking ALL letter tokens -- it is what the cross-block patch uses, and narrowing it broke every o200k-family gate until the two were separated. english MB/s: o200k 504 -> 1091, tekken 1547 -> 1507, kimi 287 -> 550. 8 gates green.
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
- `classify/` owns its own pieces instead of scattering them at the crate root: the generated `atom_tables.rs`, the `tables.rs` layout it bakes, and the three per-arch kernels (`neon.rs`, `avx.rs`, `wasm.rs`, renamed from `simd_*_classify.rs`). - `models/<name>/` — one folder per unrolled pre-tokenization regex. Models sharing a regex share a folder: o200k covers Llama-4 / gpt-oss / MiniMax-M2, cl100k covers Llama-3 / GLM-4.6 and Qwen at digit cap 1. bitmap_gen now writes classify/atom_tables.rs and emits `use super::tables::Tables;`. Regenerating produces a byte-identical file apart from that import. Root re-exports (`bitsplit::bitsplit_o200k`, ...) are unchanged, so tk-encode is untouched. 23 suites green.
- `models/<name>.rs` — a file per regex, not a folder. - `simd/` holds the per-arch kernels: `neon.rs` + `x86.rs` (block builder) and `classes.rs` (class-run boundaries), replacing the loose `simd*.rs` at the root. classify keeps its own kernels next to the tables they index. - `bitsplit/examples/` removed, along with the memchr dev-dep it needed. The numbers they produced are recorded in the PR description.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cleanup on top of #2306. bitsplit is now the only crate —
atomsplitis deleted — and everymodel family you listed routes to a byte-exact bitstream grammar.
Coverage
Every regex below was fetched from the model's live
tokenizer.json(ortokenization_kimi.py),not assumed. Six of the seven collapse onto two existing grammars:
Sequencedeepseek.rscl100k.rs\p{N}cl100k.rs(cap 1)o200k.rso200k.rsRemoved+invert, canonicalized)o200k.rs[\p{Han}]+, Han removed from the letter classes,[\r\n]*tailkimi.rstekken.rsStructure
One file per regex (models sharing a regex share a file), each unrolled against the shared
primitives in
lib.rs(scanthru,adv,to_lead,fill_to_last,digit_groups,emit).An earlier
<AUX, CONTRACTION, DIGIT_CAP>const-generic version is gone: it made every fix athree-way risk and none of the three could be read on its own.
classify+ tables + the three SIMD kernels +regexes+Spanmoved into bitsplit.fast_builder()went with them — with no second enginethere was nothing left to gate.
CharDelimiterSplitlive inbitsplit::classes.literal.rsis a bitstream (see below);memchris no longer a bitsplit dependency.Byte-exactness
#2306 shipped bitsplit with no tests —
lib.rscited asrc/bin/verify.rsthat is not in thetree.
tests/parity.rsis the gate now:crosses the 64-byte grid in every alignment. This is what actually exercises
starts[bi-1] |= patchand the
anlretraction, and it caught every bug below.#[should_panic]negative control, so the gate cannot silently stop comparing8 gates green: gpt2, cl100k, qwen, deepseek, o200k, tekken, kimi + the control.
Two findings worth calling out, both caught by the sweep:
中Qzis one token butʰABCis two — the differenceis whether an
Lappears later in the run, so nop1decides it. It is a scalar escape behinda cheap bit gate; all-lowercase and Capitalised text never pays.
\p{M}is in both the letter classes and rule 4's[^\s\p{L}\p{N}].!\u{301}ais one token,!!\u{301}ais two. Real text hits this via emoji + variation selector (U+FE0F is\p{Mn}).Throughput
cargo run --release -p bitsplit --example grammars, MB/s, split only (classify excluded):Literal (metaspace
U+2581needle, 8 MB,--example litbench): 1433 MB/s vs memmem's 1126 (1.27×).A scalar first-byte compare measured 0.64× — the SIMD first-byte bitmap is what makes it a win.
Metaspace (llama2 / gemma)
to_normalizer_and_splitalready decomposes Metaspace at load time into aMetaspaceNormalizerSpliton theU+2581delimiter. ThatSplit's search is now the literal bitstream, sometaspace runs on bitsplit end to end with no new dispatch.
Known gaps
bitsplit::classeswas moved, not ported to bitstreams. A class-run boundary isc & !(c << 1),which would delete
simd_classes.rsoutright — marked with aponytail:comment. Pure cleanup.ponytail:comment), which is most of why ittrails the others. Vectorising it mirrors the existing CJK path.
benches/(the cross-engine onig / fancy-regex / logos / pcre2 comparison) went withthe crate. The two
examples/above replace the throughput half; say the word and I'll port thecomparison bench over.
than the FSM was.
simd128build64is the upgrade path.