perf(bpe): bring the target-encode fast path onto the POC - #2318

Merged

ArthurZucker merged 19 commits into

poc/target-encode-cleanfrom

port/wip-perf

Aug 7, 2026

Commits on Aug 7, 2026

perf(bpe): hash a word once for both the fold and the cache probe
ArthurZucker
committed
d0b7750
fix(bpe): repair `tokenize_spans`, which does not compile (#2314)
ArthurZucker
authored
d64ad20
perf(bpe): pack a short word into its key instead of hashing it (#2315)
ArthurZucker
authored
473db76
Merge branch 'feat/train_encode_split' into perf/bpe-hash-word-once
ArthurZucker
authored
d00cce3
perf(bpe): put the key in the entry, so the fold probe is two loads (#2316)
ArthurZucker
authored
246d122
perf(bpe): hash once on the batched path too
ArthurZucker
committed
dcb3132
perf(pipeline): let the caller own the output buffer
ArthurZucker
committed
574ee9b
Merge remote-tracking branch 'origin/perf/bpe-hash-word-once' into bench/2306-plus-2313
ArthurZucker
committed
2ad1b9d
merge: resolve duplicate tokenize_spans (keep the cache-aware, keyed version)
ArthurZucker
committed
882af76
perf(bpe): carry pair ranks across multipass passes, and stop splicing
ArthurZucker
committed
be55698
perf(bpe): write cache hits straight at the output cursor
ArthurZucker
committed
3508f83
perf(bpe): take the target-encode word cache and vocabulary store
ArthurZucker
committed
61c82c1
perf(bpe): reserve one id per span, and key from a masked load
ArthurZucker
committed
975ed8d
Revert "perf(bpe): carry pair ranks across multipass passes, and stop splicing"
ArthurZucker
committed
3049832
strip the rationale out of the code
ArthurZucker
committed
0717165
drop the port's own rationale comments, keep the ones that were already there
ArthurZucker
committed
bb13400
Merge remote-tracking branch 'origin/poc/target-encode-clean' into port/wip-perf
ArthurZucker
committed
db6fbbc
wide 1.6.0 is yanked -> update cargo.toml to reference 1.5.0
LysandreJik
authored
dd84d58
perf(bpe): probe the cache before the fold
ArthurZucker
committed
b67245a

Navigation Menu

Uh oh!

perf(bpe): bring the target-encode fast path onto the POC - #2318

perf(bpe): bring the target-encode fast path onto the POC#2318
ArthurZucker merged 19 commits into
poc/target-encode-cleanfrom
port/wip-perf

Commits on Aug 7, 2026

perf(bpe): hash a word once for both the fold and the cache probe

fix(bpe): repair `tokenize_spans`, which does not compile (#2314)

perf(bpe): pack a short word into its key instead of hashing it (#2315)

Merge branch 'feat/train_encode_split' into perf/bpe-hash-word-once

perf(bpe): put the key in the entry, so the fold probe is two loads (#2316)

perf(bpe): hash once on the batched path too

perf(pipeline): let the caller own the output buffer

Merge remote-tracking branch 'origin/perf/bpe-hash-word-once' into bench/2306-plus-2313

merge: resolve duplicate tokenize_spans (keep the cache-aware, keyed version)

perf(bpe): carry pair ranks across multipass passes, and stop splicing

perf(bpe): write cache hits straight at the output cursor

perf(bpe): take the target-encode word cache and vocabulary store

perf(bpe): reserve one id per span, and key from a masked load

Revert "perf(bpe): carry pair ranks across multipass passes, and stop splicing"

strip the rationale out of the code

drop the port's own rationale comments, keep the ones that were already there

Merge remote-tracking branch 'origin/poc/target-encode-clean' into port/wip-perf

wide 1.6.0 is yanked -> update cargo.toml to reference 1.5.0

perf(bpe): probe the cache before the fold