bitsplit: one crate, all model grammars, atomsplit deleted - #2317

Merged

ArthurZucker merged 11 commits into

poc/target-encode-cleanfrom

poc/bitsplit-single-crate

Aug 7, 2026

Commits on Aug 7, 2026

refactor(bitsplit): absorb classify/literal/regexes, add the byte-exactness gate
ArthurZucker
committed
fd93c2c
feat(bitsplit): o200k family (llama4 / gpt-oss / minimax / tekken / kimi), 4th plane, aux streams
ArthurZucker
committed
82ce355
fix(bpe): adopt #2314's tokenize_spans verbatim
ArthurZucker
committed
9faf232
refactor(bitsplit): one file per regex, unrolled
ArthurZucker
committed
0d33e98
fix(bitsplit): o200k / tekken / kimi are byte-exact
ArthurZucker
committed
fe73719
refactor(bitsplit): split gpt.rs into gpt2.rs + cl100k.rs, add Qwen
ArthurZucker
committed
bf28b46
refactor: delete atomsplit, tk-encode runs on bitsplit
ArthurZucker
committed
bee3a3e
feat(bitsplit): literal search as a bitstream, absorbing CharDelimiterSplit
ArthurZucker
committed
a6461e9
perf(bitsplit): escape only the letter runs that need it
ArthurZucker
committed
b62d1e9
refactor(bitsplit): classify/ and models/<name>/ folders
ArthurZucker
committed
df644d6
refactor(bitsplit): file per model, simd/ folder, drop examples
ArthurZucker
committed
12dc9c5