TARGET POC: bitsplit - #2306

Open

ArthurZucker wants to merge 8 commits into

feat/train_encode_splitfrom

poc/target-encode-clean

Commits on Aug 6, 2026

perf(pre-tokenizer): bitstream splitter for the gpt2 and cl100k regexes
ArthurZucker
committed
d1ad772
perf(pipeline): give the model a whole chunk of pre-tokens at a time
ArthurZucker
committed
6fe801f

Commits on Aug 7, 2026

Merge branch 'feat/train_encode_split' into poc/target-encode-clean
ArthurZucker
authored
b13b2b9
bitsplit: one crate, all model grammars, atomsplit deleted (#2317)
ArthurZucker
authored
28fbb60
perf(bpe): bring the target-encode fast path onto the POC (#2318)

ArthurZucker
and
LysandreJik
authored
710fa22
perf(added-vocab): search the whole prefix, not just its first byte
ArthurZucker
committed
bd6560b
perf(vocab): inline the key packing, keep the real hash out of line
ArthurZucker
committed
c4e9b9a
perf(bpe): hold the write cursor and the cache across a chunk
ArthurZucker
committed
f54e490