TARGET MINIMAL RUST CRATE + .tok FORMAT - #2293

Open

ArthurZucker wants to merge 97 commits into

feat/train_encode_splitfrom

feat/tok-format

Commits on Jul 23, 2026

wip / ai: scratch pool
SBrandeis
committed
1cffe7f
scratch pool iteration
SBrandeis
committed
7556ac7
implement word cache
SBrandeis
committed
b5ef46f
lint
SBrandeis
committed
8000719
change defaults + max length + capacity bump
SBrandeis
committed
ee332b0
more sensible values
SBrandeis
committed
75f796c
rewrite without hashmap
SBrandeis
committed
d42d866
iteration
SBrandeis
committed
4106c3f
iteration
SBrandeis
committed
96a1778
fmt
SBrandeis
committed
40d37b3
review comments
SBrandeis
committed
6b26583

Commits on Jul 24, 2026

draft first commit
ArthurZucker
committed
c87102d
nits
ArthurZucker
committed
dfde37a
proper bpe only benches
ArthurZucker
committed
1436856
proper features
ArthurZucker
committed
6759ed1
update
ArthurZucker
committed
3c532af
update what we balckbox
ArthurZucker
committed
43ac1ab

Commits on Jul 27, 2026

comments
ArthurZucker
committed
062c6f8
commetn
ArthurZucker
committed
cb4123a
first drafts
ArthurZucker
committed
311eec7
fix glue
ArthurZucker
committed
330972f

Commits on Jul 28, 2026

draft table builds
ArthurZucker
committed
3449330
start mphf draft
ArthurZucker
committed
5a9cd45
impl get
ArthurZucker
committed
8e3347b
fix mphf
ArthurZucker
committed
f11a5f9
nits
ArthurZucker
committed
e0a073d
fixes
ArthurZucker
committed
8c667db
slowly but surely
ArthurZucker
committed
6ffbfba
fixes
ArthurZucker
committed
346e99e
table tests pass
ArthurZucker
committed
2227c40
create top merges
ArthurZucker
committed
12690e9
update
ArthurZucker
committed
3474720
up
ArthurZucker
committed
d66b177

Commits on Jul 29, 2026

progress in the conversion table build
ArthurZucker
committed
14c0a2f
update
ArthurZucker
committed
b1f44bc
fixes
ArthurZucker
committed
519bbcc
dman
ArthurZucker
committed
9f23a46
...
ArthurZucker
committed
795131c
eager flag was missing
ArthurZucker
committed
07efea6
update
ArthurZucker
committed
4066919

Commits on Jul 30, 2026

design a manual example
ArthurZucker
committed
33d0001
with / without thief
ArthurZucker
committed
836bcfe
remove some shit
ArthurZucker
committed
1b498e3
more simplifications
ArthurZucker
committed
b432729
single test move it
ArthurZucker
committed
b3fcd70
draft engine
ArthurZucker
committed
d542edc
remove complication
ArthurZucker
committed
1965a6c
remove one flag that was not worth it
ArthurZucker
committed
7ae223c

Commits on Jul 31, 2026

current status
ArthurZucker
committed
ec119a6
monomorphize for perfs
ArthurZucker
committed
3a7aa8f
fixes
ArthurZucker
committed
c3aad7a

Commits on Aug 3, 2026

rename
ArthurZucker
committed
abae229
start drafting the two tier merge
ArthurZucker
committed
8a616f3
todo is getting fixed
ArthurZucker
committed
f0e6031
nit
ArthurZucker
committed
5df6783
SparseFold for table!
ArthurZucker
committed
678b9f6
move stuff around
ArthurZucker
committed
27ab17b
multipass bpe
ArthurZucker
committed
4b3f37b
scratches
ArthurZucker
committed
d3a8b02
isolate pipeline bpe
ArthurZucker
committed
d0d6e41
remove unused mergemap
ArthurZucker
committed
ba35ffc
u16 is faster
ArthurZucker
committed
99e0846
add convert.rs
ArthurZucker
committed
3603372
update
ArthurZucker
committed
563ab9b
less branches
ArthurZucker
committed
e474cdb
update
ArthurZucker
committed
6fa4281
it was not worth it
ArthurZucker
committed
23542bd
finally
ArthurZucker
committed
abbb762
nits, renaming and moving files here and there
ArthurZucker
committed
088e329
renaming
ArthurZucker
committed
bd95276
fix more of the renaming
ArthurZucker
committed
06dbb96
pipeline vs legacy
ArthurZucker
committed
1fee23b
Merge remote-tracking branch 'origin/feat/train_encode_split' into perf/bpe-merge
ArthurZucker
committed
93bd4ac
first fix for the gate
ArthurZucker
committed
703e37f
update the local bench
ArthurZucker
committed
8523274
up
ArthurZucker
committed
0bde560
bit better benches
ArthurZucker
committed
11e1786
update
ArthurZucker
committed
1ce1f6c
revert unrelated changes
ArthurZucker
committed
a6bea80
more reverts
ArthurZucker
committed
47098dc
more reversion!
ArthurZucker
committed
a42edc9
cleanup
ArthurZucker
committed
9dc69e3
nits
ArthurZucker
committed
490e3f0
more clippy
ArthurZucker
committed
4d2b037
rewrite + document multipass
SBrandeis
committed
2624c43
rename
SBrandeis
committed
db1798d
perf: save a lookup when there is no merge
SBrandeis
committed
7598fb2
perf: compiler gotchas
SBrandeis
committed
ff4d576
Idiomacy:
SBrandeis
committed
e9fe3ca

Commits on Aug 5, 2026