perf(pipeline): don't stride an input that has no cut points - #2283
Merged
ArthurZucker merged 3 commits intoAug 6, 2026
Merged
Conversation
`plan()` chose `Raw` whenever the *config* exposed a stride boundary, without asking whether the *input* contains one. Punctuation-terminated CJK contains none at all: Chinese prose has no space anywhere, and a newline after `。` is not a legal cut because the punct rule ` ?[…]+[\r\n]*` absorbs it into the punct token (`NEWLINE_PREV` excludes punctuation for exactly that reason). `data/corpora/chinese.txt` is 100% such newlines -- 722 of them, every one preceded by `。`, and zero spaces. Striding then did what `stride_range`'s own doc comment warns about: "text with no boundaries at all degrades to stride 0 owning the whole input". Measured on a 4 MB doc: 513 strides tiled, **1 resolved, covering all 4199874 bytes**, after scanning the document twice looking for cuts that do not exist. 0.28x the plain serial encode, and flat across thread counts because there is only ever one work unit. `plan()` now probes the longest input for an actual cut (bounded at 32 KB, once per encode) and falls through to `Pretokenized` when there is none -- which pre-tokenizes serially and parallelises the model over span groups. That prefix runs the normal pre-tokenizer, so it is `bitsplit` wherever bitsplit is wired (gpt2, cl100k), which is what makes it cheap enough to prefer over a degenerate stride. gpt2 chinese, 14 x 4 MB docs, MB/s ours/giga -- was 1193/1243 (0.94x), now 3004/1120 (2.68x); on a cool box the same fix measured 4798/1243 (3.86x). Single thread on one doc: 260 -> 732. Token counts unchanged (23531424), and every plan was already byte-exact, so plan choice only ever trades throughput. Spaced text still probes as cuttable and takes `Raw` unchanged. `cut_exists` passes `lo = 1`, not 0: `boundary_in_window` reads one byte of left context via `block_lo - 1`, so 0 underflows to `usize::MAX` and spins forever in the char-boundary walk-back (it hangs, it does not panic, in release). Byte 0 is never a cut anyway. Written with nested `if let` rather than let-chains: this branch is edition 2018.
One 4 MB document per thread through the parallel `encode`, matching gigatoken's `hf_mt` (`encode_docs_ragged` over the same docs) in thread count, work per thread and MiB/s. 14 threads, best-of-3 interleaved, ours/giga: gpt2 english 6240/9229 .68x | code 4252/3056 1.39x | chinese 1181/1251 .94x | russian 4463/1185 3.77x llama-3 english 5075/8766 .58x | code 3380/4071 .83x | chinese 1380/1926 .72x | russian 4894/1638 2.99x Geomean: gpt2 1.35x, llama-3 1.01x. Scaling against each side's own 1-thread encode is the more useful read: english 5.4x/4.5x (giga 6.2x/6.5x), code 6.8x/4.9x (3.9x/5.2x), russian 6.4x/4.9x (2.0x/2.6x), but **chinese only 1.6x on both models** (giga 2.5x/3.8x) -- the whitespace-boundary striding limit already noted for single-document CJK, which is why gigatoken overtakes us at 14 threads on chinese despite our 1.5-1.7x single-thread lead there.
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
McPatate
approved these changes
Aug 4, 2026
ArthurZucker
commented
Aug 5, 2026
| return false; | ||
| } | ||
| let hi = text.len().min(PROBE); | ||
| let hi = (1..=hi).rev().find(|&i| text.is_char_boundary(i)).unwrap_or(1); |
Collaborator
Author
There was a problem hiding this comment.
this can probably be improved as well btw, find is not really great
ArthurZucker
merged commit Aug 6, 2026
6610330
into
feat/multi_threaded_pipeline
31 of 42 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
plan()choseRawwhenever the config exposed a stride boundary, without asking whether the input contains one. Punctuation-terminated CJK contains none: Chinese prose has no space anywhere, and a newline after。is not a legal cut because the punct rule?[…]+[\r\n]*absorbs it into the punct token —NEWLINE_PREVexcludes punctuation for exactly that reason.data/corpora/chinese.txtis 100% such newlines: 722 of them, every one preceded by。, and zero spaces.Striding then did what
stride_range's own doc comment warns about — "text with no boundaries at all degrades to stride 0 owning the whole input". On a 4 MB doc: 513 strides tiled, 1 resolved, covering all 4199874 bytes, after scanning the document twice for cuts that don't exist. 0.28× the plain serial encode, and flat at every thread count because there is only ever one work unit.plan()now probes the longest input for a real cut (bounded 32 KB, once per encode) and falls through toPretokenizedwhen there is none. That prefix runs the normal pre-tokenizer, so it'sbitsplitwhere bitsplit is wired (gpt2, cl100k) — which is what makes it cheap enough to prefer over a degenerate stride.14 × 4 MB docs, MB/s ours/gigatoken, interleaved best-of-3, quiet box:
gpt2 chinese went 0.94× → 3.82×. Single doc, one thread: 260 → 732 MB/s. Token counts unchanged (23531424) — every plan was already byte-exact, so plan choice only trades throughput. Spaced text still probes as cuttable and takes
Rawon the identical path; english measured 0.68× → 0.70×.cut_existspasseslo = 1, not 0:boundary_in_windowreads one byte of left context viablock_lo - 1, so 0 underflows tousize::MAXand spins forever in the char-boundary walk-back — it hangs rather than panicking in release. Byte 0 is never a cut anyway.Stacked on
feat/multi_threaded_pipelinebecause the plan ladder it fixes lands with #2213. Retarget tofeat/train_encode_splitonce #2213 merges.Two notes on the port from
poc/target-encode, where this was developed and measured:plan()is written with nestedif letinstead of the let-chains the original used. Same ladder, same order.examples/ab_giga_mt.rsis included so the numbers are reproducible: it mirrors gigatoken's ownhf_mtbench (one 4 MB doc per thread,encode_docs_raggedvs our batchencode, same MiB/s convention).292 lib tests + 35 integration tests pass, including a new
uncuttable_input_skips_stridingthat pins all three cases:。+newline is not a cut, uncuttable input is not strided, and Han-letter+newline and ordinary spaced text still takeRaw.