Before a large language model "thinks" about anything, it chops the input into tokens. The tokenizer's vocabulary was learned from a corpus that is overwhelmingly English, so the same meaning costs very different amounts depending on the language: running is one token in English, while its equivalent in Urdu can take four to eight tokens — sometimes fragmented down to individual bytes.
This is not a cosmetic detail. The number of tokens a language consumes drives cost (you pay per token), context window (fewer tokens fit), latency (slower generation), and quality (worse representation downstream). Build a product in English and "localize" it later, and you ship a worse, slower, more expensive experience to non-English users without ever seeing it on a dashboard.
This repository contains a single reproducible notebook — tokenization-bias.ipynb — that audits this effect for any language. It works with languages from the Mozilla Data Collective catalog — Urdu, Polish, Punjabi, Saraiki, and Balochi — and measures how badly each one gets fragmented by today's tokenizers (GPT-4o, GPT-4 / 3.5, and Llama 3), all the way through to what it costs you in dollars.
It is Layer 2 — the tokenizer — of a larger framework for understanding multilingual bias in LLMs. Tokenization is the measurable, mechanical, pre-model layer; every layer above it (training-data coverage, instruction tuning, cultural alignment, evaluation, deployment) compounds on top.
Requires Python 3.12+.
git clone https://github.com/Mozilla-Data-Collective/tokenization-bias.git
cd tokenization-bias
python -m venv .venv && source .venv/bin/activate
The notebook installs its own dependencies inline (%pip install ...), so opening and running it top to bottom is enough. To install them up front instead:
pip install tiktoken transformers matplotlib pandas anthropic datacollective
-
Parts 1–4 need no API keys. Open
tokenization-bias.ipynband run from the top. These cells use OpenAI'stiktoken(runs locally) and download the public Llama 3 tokenizer to compare token counts across languages and tokenizers. -
Parts 5–6 use real MDC corpora. To download the literary datasets, get an API key from the Mozilla Data Collective platform and add it to a
.envfile (see.env.example):MDC_API_KEY=your-api-key-here
Datasets are fetched via the
datacollectivePython library. -
(Optional) Exact Claude token counts. The bonus cell uses Anthropic's
count_tokensendpoint. SetANTHROPIC_API_KEYin your environment to run it; skip it otherwise.
The notebook is a six-part walkthrough:
| Part | What it shows |
|---|---|
| 1. What a tokenizer does | Encodes one sentence and renders each token as a colored chunk |
| 2. Same meaning, different language | UDHR Article 1 as a clean parallel corpus, tokenized side by side |
| 3. The inflation ratio | How many tokens it takes to say the same thing vs. English |
| 4. Across tokenizers | GPT-4o (o200k), GPT-4 / 3.5 (cl100k), and Llama 3 compared |
| 5. Real literary text | Tokens-per-character on actual prose extracted from MDC corpora |
| 6. What it costs in dollars | A cost model for a 1M-query/month multilingual support agent |
Each part is self-contained — swap the languages, the samples, or the models and the framework still holds.
Part 5 downloads five corpora from the Mozilla Data Collective platform:
- Rana Printers Urdu Literature Corpus
- Polish Public Domain 20th Century Literature Text Corpus
- Chishti Sons Punjabi Literature Corpus
- Saraiki Literature Corpus
- Eastern Balochi Literature Corpus
Many more low-resource languages are available on the platform — Gojri, Khowar, Hazargi, Western Balochi, and others — and can be dropped into the same audit.
- Drop in a sample from Khowar, Gojri, or Hazargi — languages essentially absent from mainstream training corpora.
- Try a different tokenizer: Gemma, Mistral, or Qwen.
- Compare two samples of the same language in different registers (literary vs. colloquial).
- Estimate the cost of running your actual product in a low-resource language.
Released under the Mozilla Public License 2.0.