perf: simd literal matching by SBrandeis · Pull Request #2312 · huggingface/tokenizers

SBrandeis · 2026-08-06T15:41:51Z

PipelineTokenizer benchmark

10 / 10 models supported — PipelineTokenizer vs tokenizers v0.23.1 (latest release) · whole corpus · ~10 kB chunks · cold caches · add_special_tokens on · 2/4-thread sweep per group

763efb2ab · 2026-08-06 17:52 UTC · Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz · 48 cores

⚠️ Release numbers are reused from a cached baseline run, and the canary (eng_Latn, re-measured by every model job) drifted +26.8% on gpt2, beyond ±10%. The runners changed since the cache was seeded; read the vs-release throughput comparisons with suspicion (allocation counts are unaffected). Any edit to fixture_bench.rs re-keys the cache and re-measures.

Decode — the release encodes each fixture's decode sample (add_special_tokens=true) and both implementations decode those SAME ids with skip_special_tokens=false, so the comparison is decode alone. MB/s counts the input bytes the ids came from, the same denominator as the encode charts.

vs base branch (16d4b9349) — per-model geomean ×speedup of this PR's PipelineTokenizer against the base branch's; regressions in red.

Work vs base (16d4b9349, allocation lane):

encode allocations: ⚠ counts changed on 45 of 150 fixtures: gemma-4/added_special_sparse +49.87%, llama-2/added_special_sparse +26.83%, gemma-4/jpn_Jpan +25.15%, gemma-4/hin_Deva +25.00%, gemma-4/rus_Cyrl +25.00%, gemma-4/tam_Taml +25.00%, and 39 more
decode allocations: ⚠ counts changed on 30 of 150 fixtures: gemma-4/cmn_Hani +33.22%, gemma-4/jpn_Jpan +33.19%, llama-2/arb_Arab +33.15%, gemma-4/agentic-traces +32.89%, gemma-4/amh_Ethi +32.87%, llama-2/tha_Thai +32.84%, and 24 more

_{allocation lane measured on: Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz · glibc 2.39}

bert-base-uncased — normalizer-heavy WordPiece · ×4.85 vs v0.23.1 · ×0.99 vs base · decode ×1.07

bert-base-uncased thread scaling (modalities)

bert-base-uncased decode thread scaling (lang)

bert-base-uncased decode thread scaling (modalities)

Memory (RSS MB, load+encode+decode): v0.23.1 7+11+0 (peak 38) · Pipeline 4+4+0 (peak 43)

Allocations (encode pass, whole corpus): v0.23.1: 159.5M allocs, 52.04 GB allocated, peak live 37 MB; Pipeline: 61.6k allocs, 0.94 GB allocated, peak live 44 MB

Allocations (decode pass, decode sample): v0.23.1: 98.5M allocs, 1.26 GB allocated; Pipeline: 98.5M allocs, 1.26 GB allocated

Fixture	Group	v0.23.1 MB/s	Pipeline MB/s	Speedup	Decode	allocs/MB	Δ base	Ids
amh_Ethi	lang	8.0	27.4	×3.43	×1.05	660 (×1,461)	×1.00	match
arb_Arab	lang	4.2	25.6	×6.14	×1.07	752 (×3,636)	×0.98	match
cmn_Hani	lang	3.7	19.3	×5.19	×1.07	855 (×4,638)	×0.99	match
eng_Latn	lang	4.4	18.4	×4.14	×1.07	670 (×3,178)	×0.97	match
hin_Deva	lang	6.4	28.0	×4.41	×1.06	642 (×2,434)	×1.00	match
jpn_Jpan	lang	4.2	26.9	×6.39	×1.07	864 (×3,342)	×1.00	match
rus_Cyrl	lang	3.6	25.1	×6.93	×1.06	752 (×4,727)	×1.01	match
tam_Taml	lang	7.0	38.1	×5.48	×1.08	644 (×2,168)	×1.01	match
tha_Thai	lang	8.2	32.5	×3.96	×1.05	622 (×1,546)	×1.02	match
added_normalized_sparse	modalities	5.2	17.7	×3.38	×1.07	68,958 (×23.29)	×0.97	match
added_special_sparse	modalities	3.9	21.3	×5.44	×1.07	89,671 (×38.89)	×1.00	match
agentic-traces	modalities	3.7	18.3	×4.92	×1.07	702 (×3,726)	×0.97	match
agentic_swe	modalities	3.8	18.9	×4.90	×1.07	755 (×3,476)	×0.97	match
code_mixed	modalities	3.8	18.7	×4.95	×1.08	776 (×3,615)	×0.98	match
math_latex	modalities	3.9	18.2	×4.65	×1.08	661 (×3,725)	×0.97	match

deepseek-v4 — deepseek 3-regex split-heavy byte-level BPE · ×15.16 vs v0.23.1 · ×1.02 vs base · decode ×10.89

deepseek-v4 decode thread scaling (lang)

deepseek-v4 decode thread scaling (modalities)

Memory (RSS MB, load+encode+decode): v0.23.1 57+0+0 (peak 75) · Pipeline 41+0+0 (peak 88)

Allocations (encode pass, whole corpus): v0.23.1: 153.2M allocs, 38.67 GB allocated, peak live 66 MB; Pipeline: 28.0k allocs, 0.23 GB allocated, peak live 74 MB

Allocations (decode pass, decode sample): v0.23.1: 15.3M allocs, 0.77 GB allocated; Pipeline: 7.4k allocs, 0.06 GB allocated

Fixture	Group	v0.23.1 MB/s	Pipeline MB/s	Speedup	Decode	allocs/MB	Δ base	Ids
amh_Ethi	lang	4.3	69.6	×16.14	×14.29	570 (×3,582)	×1.04	match
arb_Arab	lang	4.2	56.0	×13.42	×9.92	378 (×5,022)	×1.02	match
cmn_Hani	lang	3.5	47.7	×13.66	×10.04	397 (×4,758)	×1.01	match
eng_Latn	lang	3.0	68.2	×22.42	×11.84	385 (×7,925)	×1.04	match
hin_Deva	lang	5.3	88.3	×16.81	×10.54	366 (×4,352)	×1.03	match
jpn_Jpan	lang	3.9	46.0	×11.88	×10.36	418 (×3,717)	×1.04	match
rus_Cyrl	lang	4.2	45.3	×10.71	×8.35	376 (×4,319)	×1.03	match
tam_Taml	lang	5.7	51.2	×8.95	×8.26	368 (×2,674)	×1.07	match
tha_Thai	lang	5.6	23.6	×4.20	×8.94	357 (×1,476)	×1.03	match
added_normalized_sparse	modalities	5.1	92.1	×18.03	×13.55	490 (×5,070)	×1.05	match
added_special_sparse	modalities	3.8	58.3	×15.34	×9.51	391 (×8,796)	×1.01	match
agentic-traces	modalities	2.6	67.5	×26.21	×11.34	430 (×8,982)	×1.06	match
agentic_swe	modalities	2.8	78.5	×27.86	×12.71	470 (×7,846)	×1.02	match
code_mixed	modalities	2.8	62.4	×22.29	×14.28	467 (×7,449)	×0.96	match
math_latex	modalities	2.7	57.7	×21.42	×11.93	387 (×9,256)	×0.92	match

gemma-4 — byte-fallback BPE, Metaspace-style split (gemma-4) · ×2.94 vs v0.23.1 · ×0.99 vs base · decode ×1.55

gemma-4 decode thread scaling (modalities)

Memory (RSS MB, load+encode+decode): v0.23.1 286+0+0 (peak 366) · Pipeline 1+0+0 (peak 365)

Allocations (encode pass, whole corpus): v0.23.1: 29.1M allocs, 23.69 GB allocated, peak live 310 MB; Pipeline: 84.2k allocs, 2.35 GB allocated, peak live 310 MB

Allocations (decode pass, decode sample): v0.23.1: 18.9M allocs, 1.79 GB allocated; Pipeline: 25.0M allocs, 2.38 GB allocated

Fixture	Group	v0.23.1 MB/s	Pipeline MB/s	Speedup	Decode	allocs/MB	Δ base	Ids
amh_Ethi	lang	11.7	48.1	×4.11	×1.51	967 (×420)	×1.01	match
arb_Arab	lang	7.4	23.2	×3.12	×1.51	941 (×412)	×0.97	match
cmn_Hani	lang	13.2	57.4	×4.35	×1.27	1,021 (×285)	×0.98	match
eng_Latn	lang	4.0	9.0	×2.27	×1.74	959 (×586)	×1.04	match
hin_Deva	lang	9.7	27.6	×2.85	×1.69	915 (×300)	×1.02	match
jpn_Jpan	lang	13.5	44.6	×3.30	×1.25	956 (×242)	×0.98	match
rus_Cyrl	lang	7.3	19.2	×2.62	×1.47	940 (×331)	×0.97	match
tam_Taml	lang	11.3	29.3	×2.59	×1.40	921 (×209)	×0.98	match
tha_Thai	lang	12.5	37.2	×2.99	×1.33	889 (×187)	×1.00	match
added_normalized_sparse	modalities	4.4	11.1	×2.53	×1.66	1,077 (×539)	×0.99	match
added_special_sparse	modalities	4.1	11.7	×2.84	×1.83	234,101 (×6.66)	×0.92	match
agentic-traces	modalities	4.2	11.4	×2.70	×1.63	964 (×674)	×1.00	match
agentic_swe	modalities	4.3	14.0	×3.26	×1.80	1,039 (×821)	×1.00	match
code_mixed	modalities	4.3	12.7	×2.92	×1.72	1,068 (×652)	×0.97	match
math_latex	modalities	4.1	9.9	×2.43	×1.62	911 (×644)	×0.99	match

gpt2 — gpt2 ByteLevel regex · ×24.99 vs v0.23.1 · ×1.03 vs base · decode ×31.43

Memory (RSS MB, load+encode+decode): v0.23.1 21+12+0 (peak 82) · Pipeline 6+4+0 (peak 84)

Allocations (encode pass, whole corpus): v0.23.1: 243.3M allocs, 38.17 GB allocated, peak live 79 MB; Pipeline: 34.6k allocs, 0.43 GB allocated, peak live 80 MB

Allocations (decode pass, decode sample): v0.23.1: 30.9M allocs, 1.36 GB allocated; Pipeline: 6.0k allocs, 0.07 GB allocated

Fixture	Group	v0.23.1 MB/s	Pipeline MB/s	Speedup	Decode	allocs/MB	Δ base	Ids
amh_Ethi	lang	3.7	84.0	×22.54	×48.23	570 (×4,658)	×1.02	match
arb_Arab	lang	3.3	76.6	×22.92	×23.69	564 (×5,616)	×1.00	match
cmn_Hani	lang	3.2	50.2	×15.73	×30.14	572 (×4,864)	×1.02	match
eng_Latn	lang	2.9	83.6	×28.96	×22.60	386 (×8,670)	×1.02	match
hin_Deva	lang	2.5	79.7	×31.70	×37.89	549 (×7,411)	×1.00	match
jpn_Jpan	lang	3.7	42.0	×11.46	×25.34	504 (×4,683)	×1.01	match
rus_Cyrl	lang	3.7	75.6	×20.23	×24.09	564 (×5,089)	×1.08	match
tam_Taml	lang	2.3	83.2	×36.31	×58.25	553 (×10,027)	×0.99	match
tha_Thai	lang	3.0	73.8	×24.95	×51.33	533 (×6,617)	×1.01	match
added_normalized_sparse	modalities	4.1	107.9	×26.24	×30.03	490 (×6,465)	×1.12	match
added_special_sparse	modalities	3.3	73.5	×22.47	×19.44	391 (×8,582)	×1.08	match
agentic-traces	modalities	2.5	77.2	×30.60	×28.57	441 (×9,456)	×1.03	match
agentic_swe	modalities	2.7	88.6	×32.72	×38.26	491 (×8,756)	×1.01	match
code_mixed	modalities	2.4	76.3	×31.45	×34.30	496 (×8,611)	×1.01	match
math_latex	modalities	2.4	74.9	×31.85	×24.52	406 (×9,294)	×1.02	match

gpt-oss — o200k-regex byte-level BPE (gpt-oss) · ×12.10 vs v0.23.1 · ×1.31 vs base · decode ×18.56

gpt-oss decode thread scaling (modalities)

Memory (RSS MB, load+encode+decode): v0.23.1 236+0+0 (peak 310) · Pipeline 1+0+0 (peak 311)

Allocations (encode pass, whole corpus): v0.23.1: 127.8M allocs, 29.89 GB allocated, peak live 272 MB; Pipeline: 28.4k allocs, 0.23 GB allocated, peak live 272 MB

Allocations (decode pass, decode sample): v0.23.1: 14.5M allocs, 0.71 GB allocated; Pipeline: 7.4k allocs, 0.05 GB allocated

Fixture	Group	v0.23.1 MB/s	Pipeline MB/s	Speedup	Decode	allocs/MB	Δ base	Ids
amh_Ethi	lang	3.9	64.1	×16.58	×26.03	570 (×3,431)	×1.29	match
arb_Arab	lang	4.4	51.9	×11.77	×14.55	378 (×4,522)	×1.28	match
cmn_Hani	lang	4.1	27.3	×6.73	×17.24	474 (×2,360)	×1.53	match
eng_Latn	lang	3.7	63.2	×17.06	×22.06	383 (×7,145)	×1.31	match
hin_Deva	lang	6.1	73.8	×12.16	×13.16	366 (×3,680)	×1.18	match
jpn_Jpan	lang	4.5	33.3	×7.39	×18.71	475 (×1,990)	×1.38	match
rus_Cyrl	lang	4.5	40.7	×8.95	×11.85	376 (×3,878)	×1.29	match
tam_Taml	lang	5.6	39.9	×7.17	×12.87	368 (×2,372)	×1.17	match
tha_Thai	lang	5.5	20.2	×3.67	×15.16	358 (×1,345)	×1.13	match
added_normalized_sparse	modalities	5.4	93.6	×17.43	×27.59	490 (×4,723)	×1.29	match
added_special_sparse	modalities	4.6	64.7	×14.15	×16.35	391 (×7,345)	×2.10	match
agentic-traces	modalities	3.6	66.3	×18.52	×22.55	414 (×7,406)	×1.23	match
agentic_swe	modalities	3.7	79.2	×21.48	×25.07	458 (×6,524)	×1.17	match
code_mixed	modalities	3.6	76.1	×21.34	×25.09	447 (×6,828)	×1.17	match
math_latex	modalities	3.4	60.5	×17.65	×20.44	386 (×7,848)	×1.36	match

glm-5.2 — cl100k-variant regex byte-level BPE (glm-5.2) · ×14.31 vs v0.23.1 · ×1.21 vs base · decode ×15.10

glm-5.2 decode thread scaling (modalities)

Memory (RSS MB, load+encode+decode): v0.23.1 164+0+0 (peak 225) · Pipeline 1+0+0 (peak 226)

Allocations (encode pass, whole corpus): v0.23.1: 154.7M allocs, 33.00 GB allocated, peak live 212 MB; Pipeline: 29.5k allocs, 0.25 GB allocated, peak live 212 MB

Allocations (decode pass, decode sample): v0.23.1: 18.3M allocs, 0.90 GB allocated; Pipeline: 6.8k allocs, 0.05 GB allocated

Fixture	Group	v0.23.1 MB/s	Pipeline MB/s	Speedup	Decode	allocs/MB	Δ base	Ids
amh_Ethi	lang	4.3	74.0	×17.10	×26.24	570 (×3,814)	×1.29	match
arb_Arab	lang	4.6	57.8	×12.55	×10.92	384 (×4,896)	×1.27	match
cmn_Hani	lang	4.6	23.4	×5.07	×12.01	425 (×2,520)	×1.44	match
eng_Latn	lang	3.9	64.2	×16.59	×14.24	384 (×7,146)	×1.25	match
hin_Deva	lang	4.0	83.6	×20.71	×12.14	458 (×6,443)	×1.09	match
jpn_Jpan	lang	5.0	25.8	×5.11	×13.09	433 (×2,153)	×1.24	match
rus_Cyrl	lang	4.9	40.3	×8.31	×9.52	376 (×3,881)	×1.24	match
tam_Taml	lang	3.9	89.9	×22.97	×18.44	464 (×6,456)	×1.06	match
tha_Thai	lang	4.7	73.0	×15.45	×18.89	445 (×3,747)	×1.27	match
added_normalized_sparse	modalities	4.4	88.7	×20.04	×18.21	490 (×4,755)	×1.09	match
added_special_sparse	modalities	3.6	54.8	×15.00	×12.06	391 (×7,379)	×1.19	match
agentic-traces	modalities	3.4	64.5	×18.74	×15.81	422 (×7,221)	×1.17	match
agentic_swe	modalities	3.7	74.4	×19.95	×18.80	460 (×6,451)	×1.14	match
code_mixed	modalities	3.5	67.5	×19.56	×17.43	446 (×6,650)	×1.17	match
math_latex	modalities	3.4	60.2	×17.48	×16.46	390 (×7,773)	×1.25	match

llama-2 — model-bounded BPE, no pre-tokenizer · ×4.83 vs v0.23.1 · ×0.99 vs base · decode ×1.59

llama-2 decode thread scaling (modalities)

Memory (RSS MB, load+encode+decode): v0.23.1 20+6+0 (peak 63) · Pipeline 7+3+1 (peak 71)

Allocations (encode pass, whole corpus): v0.23.1: 63.0M allocs, 28.31 GB allocated, peak live 61 MB; Pipeline: 77.2k allocs, 0.56 GB allocated, peak live 72 MB

Allocations (decode pass, decode sample): v0.23.1: 35.7M allocs, 3.03 GB allocated; Pipeline: 47.4M allocs, 4.15 GB allocated

Fixture	Group	v0.23.1 MB/s	Pipeline MB/s	Speedup	Decode	allocs/MB	Δ base	Ids
amh_Ethi	lang	5.6	91.4	×16.41	×1.61	944 (×3,417)	×1.05	match
arb_Arab	lang	12.2	78.5	×6.43	×1.52	939 (×760)	×1.01	match
cmn_Hani	lang	11.0	105.0	×9.51	×1.41	943 (×1,321)	×0.97	match
eng_Latn	lang	4.5	10.8	×2.43	×1.70	919 (×667)	×0.98	match
hin_Deva	lang	14.1	113.2	×8.04	×1.52	915 (×762)	×1.01	match
jpn_Jpan	lang	16.0	132.6	×8.29	×1.43	955 (×744)	×0.99	match
rus_Cyrl	lang	9.0	29.5	×3.29	×1.51	847 (×449)	×0.99	match
tam_Taml	lang	15.3	132.1	×8.66	×1.46	921 (×844)	×1.01	match
tha_Thai	lang	19.0	128.8	×6.76	×1.45	889 (×574)	×0.99	match
added_normalized_sparse	modalities	5.2	15.0	×2.90	×1.61	979 (×695)	×0.97	match
added_special_sparse	modalities	4.7	15.2	×3.27	×1.92	184,114 (×8.86)	×0.97	match
agentic-traces	modalities	4.8	13.3	×2.74	×1.62	879 (×824)	×0.98	match
agentic_swe	modalities	4.5	13.5	×2.98	×1.84	946 (×953)	×0.98	match
code_mixed	modalities	4.7	13.7	×2.91	×1.73	974 (×909)	×0.98	match
math_latex	modalities	4.6	12.0	×2.60	×1.64	847 (×773)	×0.97	match

llama-3 — cl100k-regex byte-level BPE (llama-3), single regex · ×15.01 vs v0.23.1 · ×0.94 vs base · decode ×21.19

llama-3 decode thread scaling (modalities)

Memory (RSS MB, load+encode+decode): v0.23.1 68+2+0 (peak 90) · Pipeline 36+0+0 (peak 103)

Allocations (encode pass, whole corpus): v0.23.1: 152.3M allocs, 33.83 GB allocated, peak live 91 MB; Pipeline: 28.7k allocs, 0.24 GB allocated, peak live 92 MB

Allocations (decode pass, decode sample): v0.23.1: 17.3M allocs, 0.88 GB allocated; Pipeline: 9.6k allocs, 0.06 GB allocated

Fixture	Group	v0.23.1 MB/s	Pipeline MB/s	Speedup	Decode	allocs/MB	Δ base	Ids
amh_Ethi	lang	4.3	70.6	×16.35	×27.55	570 (×3,818)	×0.89	match
arb_Arab	lang	4.6	53.8	×11.73	×13.58	379 (×4,900)	×0.92	match
cmn_Hani	lang	4.7	23.8	×5.11	×15.33	476 (×2,372)	×0.89	match
eng_Latn	lang	4.0	65.0	×16.29	×18.02	384 (×7,149)	×0.94	match
hin_Deva	lang	4.3	109.6	×25.66	×15.85	366 (×7,163)	×1.00	match
jpn_Jpan	lang	5.0	25.4	×5.10	×20.50	433 (×2,151)	×0.91	match
rus_Cyrl	lang	4.5	37.9	×8.37	×14.27	376 (×3,985)	×0.91	match
tam_Taml	lang	3.6	93.2	×25.62	×27.54	464 (×6,460)	×0.98	match
tha_Thai	lang	4.4	55.1	×12.50	×17.91	356 (×4,118)	×0.91	match
added_normalized_sparse	modalities	4.2	97.0	×23.06	×31.89	490 (×4,760)	×0.99	match
added_special_sparse	modalities	3.4	68.2	×20.22	×21.64	391 (×7,385)	×0.96	match
agentic-traces	modalities	3.4	66.8	×19.79	×24.45	414 (×7,327)	×0.94	match
agentic_swe	modalities	3.5	78.5	×22.38	×28.97	458 (×6,458)	×0.94	match
code_mixed	modalities	3.3	67.9	×20.49	×29.44	446 (×6,650)	×0.95	match
math_latex	modalities	3.2	59.9	×18.62	×22.96	388 (×7,791)	×0.95	match

mistral-small-4 — tekken byte-level BPE, 1k added specials (mistral-small-4) · ×11.47 vs v0.23.1 · ×1.20 vs base · decode ×12.22

mistral-small-4 thread scaling (modalities)

mistral-small-4 decode thread scaling (lang)

mistral-small-4 decode thread scaling (modalities)

Memory (RSS MB, load+encode+decode): v0.23.1 151+0+0 (peak 189) · Pipeline 9+0+0 (peak 189)

Allocations (encode pass, whole corpus): v0.23.1: 135.2M allocs, 31.26 GB allocated, peak live 185 MB; Pipeline: 28.7k allocs, 0.24 GB allocated, peak live 185 MB

Allocations (decode pass, decode sample): v0.23.1: 16.6M allocs, 0.81 GB allocated; Pipeline: 7.1k allocs, 0.05 GB allocated

Fixture	Group	v0.23.1 MB/s	Pipeline MB/s	Speedup	Decode	allocs/MB	Δ base	Ids
amh_Ethi	lang	4.4	64.8	×14.83	×20.69	570 (×3,847)	×1.24	match
arb_Arab	lang	4.9	49.1	×10.05	×10.50	378 (×4,557)	×1.27	match
cmn_Hani	lang	4.6	32.2	×7.01	×13.28	476 (×2,565)	×1.44	match
eng_Latn	lang	3.8	57.5	×14.96	×12.32	388 (×7,243)	×1.30	match
hin_Deva	lang	6.2	72.8	×11.80	×9.44	366 (×3,846)	×1.17	match
jpn_Jpan	lang	5.0	35.5	×7.11	×11.41	471 (×2,080)	×1.31	match
rus_Cyrl	lang	4.7	39.2	×8.37	×9.08	376 (×4,023)	×1.24	match
tam_Taml	lang	6.2	41.5	×6.67	×9.54	368 (×2,471)	×1.16	match
tha_Thai	lang	6.2	22.4	×3.59	×10.90	358 (×1,515)	×1.08	match
added_normalized_sparse	modalities	5.5	88.6	×16.11	×14.66	490 (×4,858)	×1.10	match
added_special_sparse	modalities	4.5	64.5	×14.20	×10.69	391 (×7,459)	×1.12	match
agentic-traces	modalities	3.3	60.4	×18.18	×13.51	438 (×7,687)	×1.13	match
agentic_swe	modalities	3.4	70.1	×20.46	×14.79	471 (×6,995)	×1.13	match
code_mixed	modalities	3.4	69.8	×20.64	×14.39	465 (×6,767)	×1.13	match
math_latex	modalities	3.4	57.2	×16.99	×12.42	396 (×7,918)	×1.24	match

t5-base — Unigram + Metaspace · ×3.02 vs v0.23.1 · ×1.02 vs base · decode ×0.99

t5-base decode thread scaling (modalities)

Memory (RSS MB, load+encode+decode): v0.23.1 31+10+0 (peak 56) · Pipeline 26+14+1 (peak 94)

Allocations (encode pass, whole corpus): v0.23.1: 260.7M allocs, 35.16 GB allocated, peak live 53 MB; Pipeline: 50.7M allocs, 4.73 GB allocated, peak live 107 MB

Allocations (decode pass, decode sample): v0.23.1: 12.0M allocs, 0.64 GB allocated; Pipeline: 12.0M allocs, 0.64 GB allocated

Fixture	Group	v0.23.1 MB/s	Pipeline MB/s	Speedup	Decode	allocs/MB	Δ base	Ids
amh_Ethi	lang	6.5	23.6	×3.61	×0.96	425,906 (×6.45)	×1.04	match
arb_Arab	lang	5.0	19.4	×3.86	×0.95	563,160 (×7.24)	×1.03	match
cmn_Hani	lang	10.2	16.6	×1.63	×0.96	933,347 (×1.44)	×0.99	match
eng_Latn	lang	2.8	9.9	×3.58	×1.02	856,499 (×7.71)	×1.06	match
hin_Deva	lang	6.7	26.2	×3.92	×0.97	208,864 (×15.19)	×1.02	match
jpn_Jpan	lang	10.5	16.7	×1.59	×0.97	826,083 (×1.52)	×1.00	match
rus_Cyrl	lang	4.2	12.8	×3.06	×1.00	963,069 (×4.39)	×1.00	match
tam_Taml	lang	8.7	24.5	×2.80	×0.96	484,955 (×4.28)	×1.01	match
tha_Thai	lang	10.6	17.6	×1.67	×0.95	769,926 (×1.71)	×1.01	match
added_normalized_sparse	modalities	4.9	19.2	×3.95	×0.99	130,437 (×49.30)	×1.04	match
added_special_sparse	modalities	5.0	19.5	×3.94	×1.00	191,548 (×27.51)	×1.02	match
agentic-traces	modalities	2.9	10.0	×3.47	×1.02	1,042,076 (×5.73)	×1.03	match
agentic_swe	modalities	3.5	11.4	×3.27	×1.02	863,866 (×6.10)	×1.03	match
code_mixed	modalities	3.0	10.2	×3.36	×1.02	995,109 (×5.88)	×1.03	match
math_latex	modalities	2.7	9.7	×3.65	×1.03	890,752 (×7.35)	×1.03	match

perf: simd literal matching

763efb2

SBrandeis added the run-pipeline-bench label Aug 6, 2026

github-actions Bot removed the run-pipeline-bench label Aug 6, 2026

SBrandeis added the run-pipeline-bench label Aug 6, 2026

Navigation Menu

Uh oh!

perf: simd literal matching - #2312

perf: simd literal matching#2312
SBrandeis wants to merge 1 commit into
feat/train_encode_splitfrom
perf/literal-match

SBrandeis commented Aug 6, 2026 •

edited by github-actions Bot

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

SBrandeis commented Aug 6, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PipelineTokenizer benchmark

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

SBrandeis commented Aug 6, 2026 •

edited by github-actions Bot

Loading