perf(bpe): Last missing nits to get up to speed - #2313

Open

ArthurZucker wants to merge 10 commits into

feat/train_encode_splitfrom

perf/bpe-hash-word-once

Commits on Aug 7, 2026

perf(bpe): hash a word once for both the fold and the cache probe
ArthurZucker
committed
d0b7750
perf(bpe): pack a short word into its key instead of hashing it (#2315)
ArthurZucker
authored
473db76
Merge branch 'feat/train_encode_split' into perf/bpe-hash-word-once
ArthurZucker
authored
d00cce3
perf(bpe): put the key in the entry, so the fold probe is two loads (#2316)
ArthurZucker
authored
246d122
perf(bpe): hash once on the batched path too
ArthurZucker
committed
dcb3132
perf(pipeline): let the caller own the output buffer
ArthurZucker
committed
574ee9b
perf(bpe): u64 keys and a digest instead of storing the length
ArthurZucker
committed
900b6a4
perf(bpe): probe the cache before the fold, through the fused emit
ArthurZucker
committed
44f3b4f
perf(added-vocab): search the whole prefix, not just its first byte
ArthurZucker
committed
2ab49f9
fix(vocab): confirm a probe on the whole hash, not 32 bits of it
ArthurZucker
committed
3a2787b