From eeac45eb4da78b170b2a75152da672e97465fdd7 Mon Sep 17 00:00:00 2001 From: Dan Mattheiss Date: Thu, 18 Jun 2026 12:16:57 -0400 Subject: [PATCH] Update README.md --- README.md | 16 +++------------- 1 file changed, 3 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index 53d214c..2e435d5 100644 --- a/README.md +++ b/README.md @@ -122,16 +122,14 @@ identical before timing, every reference called through the same raw API its own benchmark uses, one back-to-back run. Three 25 MB corpora streamed from their real sources — **The Pile** (diverse), **GitHub code**, **Common Crawl** (multilingual) — across both common OpenAI encodings (throughput in **MB/s**). -quicktok is shown three ways: native C++, the Python `encode_to_numpy()` fast path, -and the plain Python `encode()` (which builds a `list[int]`): +quicktok is shown 2 ways: native C++ and the Python `encode_to_numpy()` fast path: **cl100k_base** (GPT-3.5 / GPT-4) | encoder | The Pile | Code | Common Crawl | |---|---:|---:|---:| | **quicktok (native)** | **92.8** | **114.9** | **55.6** | -| **quicktok (Python, numpy)** | **92.5** | **109.5** | **54.9** | -| quicktok (Python) | 64.3 | 75.9 | 41.6 | +| **quicktok (Python)** | **92.5** | **109.5** | **54.9** | | bpe-openai | 29.8 | 34.1 | 24.0 | | tiktoken-rs | 13.6 | 12.9 | 11.9 | | tiktoken (Python) | 12.6 | 11.8 | 10.9 | @@ -142,20 +140,12 @@ and the plain Python `encode()` (which builds a `list[int]`): | encoder | The Pile | Code | Common Crawl | |---|---:|---:|---:| | **quicktok (native)** | **73.4** | **91.7** | **39.0** | -| **quicktok (Python, numpy)** | **72.0** | **93.7** | **39.0** | -| quicktok (Python) | 57.1 | 65.4 | 33.5 | +| **quicktok (Python)** | **72.0** | **93.7** | **39.0** | | bpe-openai | 27.4 | 32.4 | 22.5 | | tiktoken-rs | 19.9 | 18.9 | 14.8 | | tiktoken (Python) | 17.3 | 16.6 | 13.4 | | TokenDagger | 8.9 | 9.9 | 7.9 | -`encode_to_numpy()` returns a `uint32` array directly, skipping the per-token -Python-list marshalling — so from Python it runs at **near-native speed** (~3× over -bpe-openai, ~7× over tiktoken). Absolute MB/s is machine- and thermal-dependent -(~15% swing on M1); the same-run ratios are the stable signal. (rs-bpe is omitted -from this run — no Python 3.14 wheel; it's a binding over the same `bpe` crate as -the bpe-openai row.) - **Reproduce these tables:** `make bench-compare` — see [bench/README.md](https://github.com/dmatth1/quicktok/blob/main/bench/README.md).