feat: multi threaded pipeline - #2213

Draft

McPatate wants to merge 16 commits into

feat/train_encode_splitfrom

feat/multi_threaded_pipeline

Commits on Jul 20, 2026

feat: impl pipeline multi-threading
McPatate
committed
c6d306f
feat: introduce parallelplan strategy
McPatate
committed
22033b3
refactor: remove unused `add_special_tokens`
McPatate
committed
61a82cd
chore: cleanup pipeline bindings
McPatate
committed
f866644
chore: cleanup pipeline benchmark
McPatate
committed
addabaa
fix: update encode API
McPatate
committed
3765e0a

Commits on Jul 21, 2026

feat: add special tokens & normalization chunking
McPatate
committed
1c0f9e5
chore: cleanup
McPatate
committed
b644aba

Commits on Jul 22, 2026

chore: more cleanup
McPatate
committed
8581dfa

Commits on Jul 23, 2026

chore: harmonize naming for segments and parallel splitting
McPatate
committed
8aec88b
chore: docs conciseness attempt
McPatate
committed
e52925c
perf: reduce allocations + avoid false sharing
McPatate
committed
2f0e387
refactor: remove channel because of immense contention
McPatate
committed
502347f
chore: trim docs
McPatate
committed
8596486

Commits on Jul 24, 2026

chore: address comments
McPatate
committed
852c82a

Commits on Aug 6, 2026

perf(pipeline): don't stride an input that has no cut points (#2283)
ArthurZucker
authored
6610330