feat(bindings): Python bindings rework, plugging PipelineTokenizer - #2229

Draft

SBrandeis wants to merge 19 commits into

feat/train_encode_splitfrom

feat/pipeline-python-api

Commits on Jul 23, 2026

decode bench + oracle + rewrite encode oracle
SBrandeis
committed
64d5586
docs iter
SBrandeis
committed
9ad0757
Apply suggestions from code review
SBrandeis
committed
8d2452d
tokenizers_pipeline
SBrandeis
committed
5e8ca6f
iteration
SBrandeis
committed
8f8631a
render python compare graph
SBrandeis
committed
fcd5c82
fold in existing package
SBrandeis
committed
ea3fb02
iteration
SBrandeis
committed
3f5bb55
macos linker config
SBrandeis
committed
87a5608
async + free-threaded python
SBrandeis
committed
e88445b
comments
SBrandeis
committed
d484bdd
don't test thread scaling on macos
SBrandeis
committed
259c52a
iteration
SBrandeis
committed
94eb3b6
iteration
SBrandeis
committed
68d6dc0
rename encode -> encode_ids
SBrandeis
committed
d888aa2
Encoding class
SBrandeis
committed
4a17bff
add fixme
SBrandeis
committed
520ca75
post-processor
SBrandeis
committed
1804a0b
feat(bindings): e2e and golden tests for Python bindings (#2235)

SBrandeis
and
claude
authored
4162de8