TARGET POC: bitsplit - #2306
Open
ArthurZucker wants to merge 6 commits into
Open
Conversation
The GPT regexes are a scan with one unpredictable branch per token. `bitsplit` decides them from bitmaps instead: build a few `u64` masks over the bytes, then take token starts 64 bytes at a time with register ops rather than a branch per character. A new crate, wired in behind two match arms in `Split::pre_tokenize`. It only takes over where its bitstream build is SIMD -- `bitsplit::fast_builder()` -- because the portable builder is slower than the FSM it replaces, so every other target and every other regex keeps the existing atomsplit FSM. cl100k is gated on the standard digit cap of 3, the only variant with a bitstream splitter. `classify_into_spans_bits` is the `classify_into_spans` equivalent for a splitter that works off bitstreams: it hands the splitter two `u64` bitmaps alongside the tags. 367 tests pass, including the pre-tokenizer oracles that hold the FSM and the bitstream splitter to the same spans as the `regex` crate.
The pipeline has the entire span list before the model runs, so handing the spans over one at a time bought nothing and cost a virtual call, a slice, a `Result` and an output capacity check per pre-token -- on English, one round trip per ~5 bytes. `Model::tokenize_spans` takes them all. Its default is the loop it replaces, so a model only overrides it when it has per-chunk work to hoist; BPE does, and there the scratch destructuring and the output reservation move out of the loop. Slicing the chunk per pre-token also re-ran a UTF-8 boundary check on spans the pre-tokenizer had already cut on char boundaries. Ported onto #2241's `model.rs` rather than cherry-picked from #2304: that PR was written against the base before #2241 landed and restructured this file, so its patch no longer applies. Same change, current structure -- and the fold fast path it now sits beside came in with #2241. 367 tests pass.
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
* refactor(bitsplit): absorb classify/literal/regexes, add the byte-exactness gate
bitsplit had no tests at all -- lib.rs pointed at a `src/bin/verify.rs` that is not
in the tree. Add `tests/parity.rs` first, so the rest of the cleanup has a gate:
oniguruma as the oracle (deepseek composed as HF applies it, three Isolated splits),
plus the block-phase sweep that is the only thing that actually exercises the
cross-block carries and `starts[bi-1] |= patch`. A `#[should_panic]` negative control
keeps the gate honest. All three existing grammars pass.
Then move the classify cluster (classify, atom_tables, tables, the three SIMD
kernels), `regexes` and `literal` into bitsplit, and put `Span` in its lib.rs.
atomsplit is now a shim over bitsplit holding only the scalar FSMs; it goes away
once every caller is rewired.
Also resolves a bad merge already on this branch: tk-encode/src/models/bpe/model.rs
carried two `fn tokenize_spans`, so tk-encode did not compile at all. Kept the
fold_id/symbols mechanics (the ones matching BpeScratch and merge_word's arity) and
restored the word-cache logic the other side had, mirroring tokenize_pipeline.
* feat(bitsplit): o200k family (llama4 / gpt-oss / minimax / tekken / kimi), 4th plane, aux streams
Adds `o200k.rs`, const-generic over <AUX, CONTRACTION, DIGIT_CAP>. AUX doubles as the
variant selector because the two text-derived streams are mutually exclusive: the
o200k/tekken line needs `/` for rule 4's `[\r\n/]*`, kimi needs `\p{Han}` for its
leading arm and has a plain `[\r\n]*` tail.
Kernel changes:
- `Blk` gains `p3`. o200k needs 9 tag classes and code 7 is reserved -- both SIMD
kernels find continuation lanes by testing `lut[tag] == 7` -- so "other" cannot be
the leftover code. Const-gated off for the 3-plane grammars.
- `Blk.cjk` generalised to `Blk.aux` with a const AUX selector (none/cjk/slash/han).
- `digit_groups::<CAP>` lifted into lib.rs: the `\p{N}{1,CAP}` block was duplicated
verbatim in cl100k and deepseek, and hardcoding 3 is what blocked Qwen.
The case split is a scalar escape, not bit algebra: `[UC]*[LC]+|[UC]+[LC]*` has no
local form (`中Qz` is one token, `ʰABC` is two -- the difference is whether an L
appears LATER in the run). The bit half computes a cheap gate instead, so
all-lowercase and Capitalised text never pays for it.
o200k/tekken/kimi parity is #[ignore]d with the reason: \p{M} is in BOTH the letter
classes and rule 4's class, and that interaction is not modelled yet. The other four
gates (gpt2, cl100k, deepseek, negative control) stay green.
* fix(bpe): adopt #2314's tokenize_spans verbatim
The duplicate-tokenize_spans repair I wrote landed upstream as #2314 on
feat/train_encode_split, which is NOT an ancestor of poc/target-encode-clean --
hence the stale base. Take the upstream text so the two cannot diverge; it also
documents why the fold must be probed before the cache.
* refactor(bitsplit): one file per regex, unrolled
Drop the <AUX, CONTRACTION, DIGIT_CAP> const generics. Folding o200k, tekken and kimi
into one parameterised grammar meant every fix was a three-way risk and nothing could
be read on its own. Now: o200k.rs (llama4 / gpt-oss / minimax), tekken.rs, kimi.rs --
each unrolled against the shared primitives in lib.rs, each independently debuggable.
The small scalar helpers (member, run_end, ws_tail, letter_match) move to lib.rs; they
are genuinely shared and copying them per file would be worse.
kimi is byte-exact. o200k/tekken still carry two cross-block bugs.
* fix(bitsplit): o200k / tekken / kimi are byte-exact
Three real bugs, all in the interaction between rule 4's `[\r\n/]*` tail and everything
around it. `/` is in BOTH the `+` body and the tail, which cl100k never has to deal with
(its tail is newlines, and a newline is not "other"):
- One tail run can collect SEVERAL markers (`!\n/\n`: the `\n` after `!` and the `\n`
after `/`). `nl_e - nl_m` spans only from the last one -- that is exactly what
`fill_to_last` exists for.
- A char the tail absorbed is not part of the `+` body, so an "other" after it opens a
fresh run (`<emoji>\r\n/#` = tail, then `#` starts again). Needs a `prev_absorbed` carry
for the block edge.
- ...but a char INSIDE the tail must not open one either, or `osf` picks up the `/` and
the `[^\r\n\p{L}\p{N}]?` prefix rule then eats the letter start after it (`#\r\n/aA`
came out as one token instead of `a` + `A`).
Plus: re-flag the last letter token whenever the escape trigger fires, since the trigger
can land in a later block than the token it belongs to.
Gate: oniguruma oracle, block-phase sweep, 4000 fuzz strings (10000 run locally, clean).
All 7 grammars green -- gpt2, cl100k, deepseek, o200k, tekken, kimi + negative control.
* refactor(bitsplit): split gpt.rs into gpt2.rs + cl100k.rs, add Qwen
One file per regex, finishing the split. cl100k's digit cap becomes a plain argument
rather than a const generic -- 3 = cl100k / Llama-3 / GLM-4.6, 1 = Qwen. cl100k was
still running its own inline hardcoded-3 digit block; it now shares digit_groups with
o200k, which is what made the cap knob real.
8 gates green: gpt2, cl100k, qwen, deepseek, o200k, tekken, kimi + negative control.
* refactor: delete atomsplit, tk-encode runs on bitsplit
The regex FSMs are gone -- every recognized pattern now routes to the byte-exact
bitstream grammar. `fast_builder()` went with them: there is no second engine left to
route to, so the gate had nothing to gate.
- `GptFsm` -> `Grammar`, one variant per distinct regex, with the Kimi key added. The
cl100k digit cap now actually reaches the splitter (Qwen routes to `bitsplit_qwen`).
- The class-run family and `CharDelimiterSplit` move to `bitsplit::classes`.
- Workspace members drop atomsplit; bitmap_gen writes bitsplit/src/atom_tables.rs;
the CI workflow is renamed.
Whole workspace green: 8 bitsplit gates, 356 tk-encode unit tests, the bpe/decode/
pipeline oracles.
* feat(bitsplit): literal search as a bitstream, absorbing CharDelimiterSplit
Replaces memmem with the same shape as the grammars: a `u64` match bitmap per 64-byte
block, walked with `trailing_zeros`. The first needle byte is compared in SIMD (the
existing `mm64` fold on NEON, `movemask` on SSE); the remaining bytes only ever run on
the survivors, so a 3-byte needle costs one vector pass plus a handful of scalar checks.
This is what puts metaspace (llama2 / gemma) fully on bitsplit: `to_normalizer_and_split`
already decomposes Metaspace at load time into a MetaspaceNormalizer + a `Split` on the
`U+2581` delimiter, and that Split's search is now a bitstream.
Measured on the metaspace needle over 8 MB (examples/litbench):
scalar first byte 727 MB/s 0.64x memmem <- not good enough
SIMD first byte 1433 MB/s 1.27x memmem
memchr is no longer a bitsplit dependency.
* perf(bitsplit): escape only the letter runs that need it
The gate was "any trigger in this block flags every letter token in it", so a single
`'s` in a paragraph dragged every word through the scalar pass. Narrow it with a reverse
fill over the letter stream: a run needs the escape only if an interior upper (or the
letter before a contraction apostrophe) sits at-or-after the run start.
`last_lt` keeps tracking ALL letter tokens -- it is what the cross-block patch uses, and
narrowing it broke every o200k-family gate until the two were separated.
english MB/s: o200k 504 -> 1091, tekken 1547 -> 1507, kimi 287 -> 550. 8 gates green.
* refactor(bitsplit): classify/ and models/<name>/ folders
- `classify/` owns its own pieces instead of scattering them at the crate root: the
generated `atom_tables.rs`, the `tables.rs` layout it bakes, and the three per-arch
kernels (`neon.rs`, `avx.rs`, `wasm.rs`, renamed from `simd_*_classify.rs`).
- `models/<name>/` — one folder per unrolled pre-tokenization regex. Models sharing a
regex share a folder: o200k covers Llama-4 / gpt-oss / MiniMax-M2, cl100k covers
Llama-3 / GLM-4.6 and Qwen at digit cap 1.
bitmap_gen now writes classify/atom_tables.rs and emits `use super::tables::Tables;`.
Regenerating produces a byte-identical file apart from that import.
Root re-exports (`bitsplit::bitsplit_o200k`, ...) are unchanged, so tk-encode is untouched.
23 suites green.
* refactor(bitsplit): file per model, simd/ folder, drop examples
- `models/<name>.rs` — a file per regex, not a folder.
- `simd/` holds the per-arch kernels: `neon.rs` + `x86.rs` (block builder) and
`classes.rs` (class-run boundaries), replacing the loose `simd*.rs` at the root.
classify keeps its own kernels next to the tables they index.
- `bitsplit/examples/` removed, along with the memchr dev-dep it needed. The numbers
they produced are recorded in the PR description.
* perf(bpe): hash a word once for both the fold and the cache probe `tokenize_pipeline` hashed every fold-missing pretoken twice: `fold_id` hashed it for the vocabulary's MPHF, then `WordCache::lookup` hashed the same bytes again for its home slot and tag. `BucketVocabStore` and `WordCache` seed `ahash` with the same four constants, so the second pass recomputed a value the first had already produced. Share it: `hash_word` exposes the value, `get_bytes_foldable_hashed` and `lookup_hashed` take it, and the model computes it once per word. Verification is untouched. The vocabulary still compares the entry's bytes to the query in full, and the cache still compares its key, so ids cannot change -- only the duplicated hash goes away. A word over fifteen bytes still pays the cache's second, independently seeded discriminant hash, which is what makes its key 127 bits rather than 64. The sharing is only sound while both sides seed identically, so a test pins them together. Re-seeding either would leave the cache placing a word under one hash and looking it up under another: no wrong ids, but every lookup would miss and the cache would quietly stop working. * fix(bpe): repair `tokenize_spans`, which does not compile (#2314) #2304 added `PipelineBPE::tokenize_spans` against the model as it stood then. #2241 replaced the merge engines and #2310 dropped `ignore_merges`, and because the two landed on separate branches the merge produced a `feat/train_encode_split` that does not build: error[E0425]: cannot find type `Span` in this scope error[E0026]: struct `BpeScratch` does not have fields named `merge_queue`, `skip`, `word` error[E0027]: pattern does not mention fields `symbols`, `queue` error[E0609]: no field `ignore_merges` on type `&PipelineBPE` error[E0061]: this method takes 3 arguments but 4 arguments were supplied Bring the batch loop back in line with `tokenize_pipeline`: destructure `{ symbols, queue, word_cache }`, run the fold, and call the current `merge_word(sequence, symbols, queue)` followed by `unmap`. The fold has to stay ahead of the cache probe, as it is in `tokenize_pipeline`. A word that is a foldable vocabulary entry is answered in one probe and never enters the cache; probing the cache first would fill it with words the fold already serves for free, and the two paths would disagree about its contents. `tokenize_spans` overrides a trait method whose default is the `tokenize_pipeline` loop, so the two can drift without anything failing to build -- that is how this got in. Add a test that runs thousands of spans through one chunk (repeats, so the cache fills and hits; folded words; merged words; punctuation runs; multi-byte scripts; a long unbroken run) and compares the ids to the legacy reference. * perf(bpe): pack a short word into its key instead of hashing it (#2315) A pretoken is short. English averages 4.83 bytes of it, code 4.08, and the `<|...|>` shapes in `added-special-dense` 2.29. Running aHash over that is most of what the fold probe costs, and it buys nothing: the vocabulary compares the entry's bytes anyway, so the hash only has to spread well enough for the MPHF to separate keys. For a word of seven bytes or fewer, pack the bytes and the length into a `u64` and mix them with one multiply. Seven, so the length still fits in the top byte, which is what keeps `"ab"` from colliding with `"ab\0"`. Longer words keep aHash, which mixes the length in itself. `word_hash` is now the one definition. `BucketVocabStore::build`, every probe, and the word cache's placement all go through it, so a pretoken probed in both tables is hashed once for the pair and the two cannot drift apart. The per-struct `RandomState` goes away with it: consistency came from carrying the hasher around, and now it comes from there being a single function. Verification is unchanged and stays exact -- the vocabulary still compares the entry's bytes to the query in full, the cache still compares its 128-bit key. Nothing verifies with `mix`, which is why it does not have to be a strong hash. Dropping the mixing altogether does not work: packed short keys share their high bytes and MPHF construction fails with "indistinguishable hashes in bucket". Note this is why `WordCache::lookup` has to call `placement_hash_of` rather than a hasher of its own: the `debug_assert` in `lookup_hashed` caught exactly that mistake while this was being written. * perf(bpe): put the key in the entry, so the fold probe is two loads (#2316) The fold probe was three dependent loads: the MPHF pilot, the entry, then the byte slab to compare the token against the query. The word cache does the same job in two, and the reason is layout, not luck -- its key lives in the slot it verifies, so nothing else has to be read. Give the vocabulary the same shape. `Entry` becomes `{ key, id }`, and `(start, len)` moves to a parallel `spans` array that only the reverse lookup and enumeration touch. A probe is now pilot + entry. Verification stays exact. A word of `INLINE_KEY_BYTES` or fewer has a key that *is* its bytes and its length, so comparing keys is proof of identity and the slab is never read. A longer word keys by aHash, which is not proof, so it still confirms against the slab -- the load it was paying anyway. So the saving lands exactly on the short pretokens that are the gap (english averages 4.83 bytes, code 4.08, `added-special-dense` 2.29) and nothing gives up the never-wrong guarantee. `LEN_TAG` now biases the length by one. A non-minimal MPHF returns padding slots, whose `Entry::default()` key is 0, and the probe rejects those with the same single compare it uses for everything else -- which only works while no real word can key to 0. The empty word keyed to exactly that before the bias. `key_and_hash` returns both halves so neither is recomputed: the model runs it once per word and hands the key and the hash to the fold probe and the hash to the cache. * perf(bpe): hash once on the batched path too `pipeline.rs` encodes through `tokenize_spans`, not `tokenize_pipeline`, and `tokenize_spans` was still hashing each word twice: `fold_id` for the vocabulary, then `cache.lookup` for the cache. Everything this PR does was landing only on `tokenize_pipeline`, which the encode loop does not call. Run `key_and_hash` once per word and hand the pair to `fold_id_keyed` and the hash to `lookup_hashed`, as `tokenize_pipeline` already does. `fold_id` had exactly one caller and folded into `fold_id_keyed` with it, taking a stale `#[allow(dead_code)]` with it. `the_batched_path_matches_the_reference` covers the path: ids compared against the legacy reference over thousands of spans in one chunk. * perf(pipeline): let the caller own the output buffer `encode_generic` sizes a fresh `Vec` from the input length -- a guess -- and hands back a new allocation on every call. A caller encoding many inputs (a batch, a server loop, a benchmark) can reserve once and `clear()` between calls instead: fewer allocations, and no first-touch of the token array each time. Split it: `encode_generic_into` takes `&mut Vec<PipelineToken>`, and `encode_generic` becomes the allocating wrapper, so nothing existing changes. Measured on identical code with both forms available, tokbench gpt2, 29 cells against gigatoken: 0.9233x allocating vs 0.9536x reusing -- ~3% of geomean throughput, ~6% on english (3.188 -> 3.001 ns/B). * merge: resolve duplicate tokenize_spans (keep the cache-aware, keyed version) The merge of #2313 into #2306 left two `tokenize_spans` definitions: git took both sides textually because they landed in different places. The stale one is #2306's, predating the word cache -- it destructures `BpeScratch { symbols, queue }` with no `word_cache` and calls the removed `fold_id`. Dropped it; kept the version that folds, probes the cache, and hashes each word once. * perf(bpe): carry pair ranks across multipass passes, and stop splicing Two changes to the multipass engine, ported from the target-encode work. **Ranks carried across passes.** A pass used to re-look-up every pair it walked over, so the passes summed to O(n^2) table lookups -- measured at **41.9 per merged word**. Only the pairs touching a merge's product actually change, so `ranks[i]` (the value of the pair `(symbols[i], symbols[i+1])`) is now seeded by `convert_multipass` -- which already looks every pair up, so seeding costs one store per pair and no extra lookup -- and carried. A pass copies the ranks it did not invalidate and pays `get_value` **twice per merge** instead of once per symbol. Finding the next target is then a scan of `ranks` with no lookups at all. `prods` holds the matching product ids, kept apart so the search array stays a dense `u32` of ranks alone. **No memmove per merge.** A merge used to splice: write the product, then `copy_within` symbols, ranks and products to close the gap -- three memmoves on every merge. Instead the word carries a `live` bitmap of which slots still hold a symbol and a merge clears one bit; "previous live" and "next live" are `leading_zeros`/`trailing_zeros`. The `MAX_MP = 24` bound is what makes this work: it puts the live set in one `u64`. Dead pair slots hold `u32::MAX`, which is also "does not merge", so the minimum search skips them for free. The superseded sweep machinery (`MergeState`, `merge_once`, `batch_merging_is_safe`, `NOT_LEGAL`) goes with it -- the batching those implemented is subsumed by carrying ranks. Byte-exact: `the_proven_fold_never_changes_the_ids` and `the_batched_path_matches_the_reference` both compare ids against the legacy reference. * perf(bpe): write cache hits straight at the output cursor A cache hit went through the tag row -- one load of 16 control bytes, a SIMD compare, then the slot -- and handed back a slice the caller walked. Both are avoidable for the common case, a word cached in its own home slot with at most three ids. `probe_emit_hashed` reads the home slot directly and stores all `MAX_INLINE_IDS` lanes unconditionally at a `*mut u32` the caller supplies, so the line is touched once and the ids never become a slice. Lanes past `ids_len` are dead: the caller advances its cursor by `ids_len` only, so the next word overwrites them or the final `set_len` cuts them off. A spilled or off-home slot falls back to the window walk without re-keying the word (`lookup_placed`, split out of `lookup_hashed`). `tokenize_spans` keeps a raw cursor and one capacity check per word covering both the fold's single write and the probe's lanes, and reserves `2 * spans.len() + MAX_INLINE_IDS` up front -- 92% of english pre-tokens are one id and 98% at most two, so the old `spans.len()` was a lower bound that made the buffer grow, and memcpy what it held, partway through most chunks. Deliberately NOT taken from the source branch: its `LookupKey` is a `u64`, which makes a hit on a word over seven bytes a 2^-64 proposition. This keeps the 128-bit key, so a hit stays exact for words up to fifteen bytes as before; only the emit is fused. Byte-exact: `the_batched_path_matches_the_reference` drives thousands of spans through this path, cache hits included, and compares ids to the legacy reference under debug assertions. * perf(bpe): take the target-encode word cache and vocabulary store u64 packed keys throughout, digest verification in the vocabulary store, and the pipelined probe helpers (probe_slot/entry_at/resolve_foldable). Accepts the exactness trades deliberately: a cache hit on a word over seven bytes is 2^-64, and an out-of-vocabulary pretoken can be mistaken for a vocabulary token at 2^-32, where both were previously impossible. * perf(bpe): reserve one id per span, and key from a masked load Two corrections to match the target-encode loop. `output.reserve(spans.len() + MAX_INLINE_IDS)`, not two apiece. Two was measured worse: the allocating entry point sizes its buffer at `len/4`, about one id per span, so asking for two forced a reallocation on every call that would not otherwise have happened. `key_and_hash_readable`: the span lies inside `chunk`, so everything up to the chunk's end is readable and a short word's key is one unaligned masked load instead of a head/tail stitch. Not done, and deliberately: wiring the pipelined probe (`probe_slot`/`entry_at`/ `resolve_foldable`). Those helpers exist but the source branch does not use them, having measured every version slower -- staging eight at a time 0.968x, carrying the next key a word early with a `prfm` 0.96x, carrying the probe answer a word early 0.95x, pairing two words 0.90x. The path is bound by instruction count, not latency. * Revert "perf(bpe): carry pair ranks across multipass passes, and stop splicing" Measured, and it does not pay: **+0.7% geomean** over tokbench's 29 gpt2 cells, inside the +-0.8% noise floor (median of four interleaved runs, five checksum-distinct binaries, ratio taken against gigatoken inside each run so it is immune to position drift). It is also lopsided rather than uniformly small: chat-llama3 1.15x, agentic-tools 1.14x, chat-deepseek 1.12x, code 1.05x, against added-normalized-dense 0.85x, hindi 0.94x, dense 0.96x. Taking table lookups from 41.9 per merged word to 2 per merge sounds decisive and is not, for an arithmetic reason: it only touches the pretokens that actually merge, which is ~8% on english. The other 92% never enter the engine, so the whole change is bounded by a small slice of the model phase. Not worth ~170 lines of engine rewrite plus two extra scratch buffers. The rest of the stack -- the fused probe, u64 keys, the digest store -- is unaffected and stays. * strip the rationale out of the code Comment blocks recording measured dead-ends, alternatives tried and their numbers belong in the PR, not in the source. Dropped every non-doc comment except `SAFETY` (load-bearing for the unsafe blocks), collapsed each doc block to its first line, kept doctests, and removed the batched-path test I had added. Net effect on the diff against this POC: +543/-227 before, +378/-442 now -- 64 lines fewer than the base rather than 300 more. * drop the port's own rationale comments, keep the ones that were already there The measured-dead-end essays and alternatives-tried notes this port added belong in the PR, not the source: 167 comment lines removed across word_cache, bucket_vocab_store and model. Every comment that existed in the base is preserved verbatim. The 33 base comment lines that no longer appear are the ones whose subject the port deleted -- `PLACEMENT_HASHER` and `DISCRIMINANT_HASHER` (gone with the u64 key), "the hasher is also stored on the struct" (the field is gone), `entries[slot] -> (offset, length, id)` (an entry is now `(digest, id)`), and `fold_id`'s doc (folded into `fold_id_keyed`). `SAFETY` comments and doctests are untouched. * wide 1.6.0 is yanked -> update cargo.toml to reference 1.5.0 * perf(bpe): probe the cache before the fold `tokenize_spans` ran `fold_id_keyed` -- an MPHF probe, a pilot load plus a dependent entry load into the whole vocabulary -- ahead of `probe_emit_keyed`, which is one load of the home slot and an unconditional store of its lanes. The expensive probe went first and answered only the words that are their own vocabulary entry, while every word the cache was about to serve paid it for nothing. On a warm cache that is nearly all of them. The cache now goes first and the fold answers the miss, where it still beats running the merge engine. A folded word is inserted, so its second and later occurrences come off the cache instead of re-probing the vocabulary. The order follows whichever probe is cheaper, and here the fused emit already made that the cache. On the branch behind #2313, where `900b6a48`'s digest store makes the fold cheap and the cache is still reached through `lookup_keyed`, the same reordering measures 0.951 -- so it is the relative cost that decides, not the order itself. ab_giga, 4 MB, single thread, warm, median of 10 rotated rounds interleaved against this branch's head with the LLC evicted between binaries. MB/s before -> after: gpt2 english 1128 -> 1182 code 609 -> 611 dense 1534 -> 1621 chinese 882 -> 902 hindi 544 -> 595 thai 617 -> 654 korean 584 -> 610 russian 708 -> 767 greek 670 -> 710 arabic 621 -> 676 llama-3 english 1090 -> 1141 code 714 -> 730 dense 1412 -> 1490 chinese 914 -> 918 hindi 835 -> 810 thai 932 -> 1003 korean 798 -> 866 russian 898 -> 968 greek 870 -> 936 arabic 904 -> 984 warm geomean 1.053, cold 1.039. Against c7ae7f4 on the same box this takes the branch from 0.925 to 0.982. Reserving two ids per span instead of one, which c7ae7f4 does, measures +0.24% here -- inside the +-0.8% geomean noise floor -- so `975ed8df`'s one-per-span reservation stays. Byte-exact: token counts unchanged on all 20 model x corpus pairs and equal to c7ae7f4's. `prove_fold` only sets the bit for an entry that merging its own text reproduces, so a folded word and a merged word give the same ids; the reorder moves which path answers, not what it answers. 370 tests pass. --------- Co-authored-by: Lysandre Debut <lysandre@huggingface.co>
The single-bucket scan looked for `prefix[0]` and rejected each candidate by hand, restarting `memchr` on `&bytes[pos + 1..]` every time. One byte of a long needle is a poor filter on text that is dense in that byte and holds no match: `<|endoftext|>` over a corpus full of `<|xs0|>` stops at every `<` and dies at the third byte, and every restart pays memchr's prologue again. Measured over the same one-bucket vocabulary in one process, so the two arms share a build: on a 9.8%-`<` corpus that matches nothing, scanning the first byte cost 0.58 ns/B against 0.12 for `memmem`, which picks a *rare* byte of the needle instead -- 4.3x. Where candidates are already sparse it costs at most 0.015 ns/B (english 0.009 -> 0.024), so it trades a little on inputs where the scan is already free for a lot on the inputs where it is not. `nibble_mask_match` already avoids this for two or more buckets, where it is called the restart penalty; the one-bucket path never got it. `find_iter` yields the same positions in the same order, so the leftmost match is unchanged and `match_fast` still confirms the length sub-list.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.