tok: o200k pre-tokenizer support, auto-detected, with download-free test coverage#330
Open
nbeerbower wants to merge 3 commits into
Open
tok: o200k pre-tokenizer support, auto-detected, with download-free test coverage#330nbeerbower wants to merge 3 commits into
nbeerbower wants to merge 3 commits into
Conversation
Inkling ships an o200k-family tokenizer (case-aware Split regex, GPT-4o
lineage) rather than cl100k. tok_load now detects the family from the
pattern itself (\p{Lu} appears only in the o200k regex) so GLM behavior
is untouched, and encode dispatches to a new pretok_chunk_o200k that
replays the regex engine's backtracking order exactly: greedy optional
prefix, maximally-greedy uppercase run given back until the lowercase
run can match, contractions attached to letter runs, \p{N}{1,3}, and
the [\r\n/]* punctuation tail.
tok_unicode_o200k.h adds the two range tables the new classes need
(Lu+Lt and Lm+Lo+M), generated from Python unicodedata.
Validated against HF tokenizers on 357 adversarial strings (case
transitions, contractions, CJK, combining marks, emoji + modifiers,
zero-width chars, 300 mixed-charset fuzz cases): 357/357 identical.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
tests/tok_o200k_tiny.json is a synthetic byte-level BPE (274 vocab, a
few KB) whose Split regex is the o200k pattern; expected ids in
tok_o200k_cases.txt were generated by HF tokenizers on that same file.
test_tok_o200k (in TEST_BINS) scores 40/40 encode + 40/40 round-trip:
case-transition splits, contractions, digit groups, the [\r\n/]* tail,
whitespace branches, CJK/Greek/Cyrillic, added-token atomicity.
The cl100k path is untouched by construction — dispatch requires
\p{Lu} in the tokenizer's own Split pattern, which cl100k lacks — and
stays covered by the GLM oracle (verified on this branch: 32/32).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
MinGW's UCRT has no getline; fixed-buffer fgets with CRLF trimming reads the same case file everywhere. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This carves the tokenizer change out of #312 so it can be reviewed and verified on its own — it's the part of that PR you said you'd check hardest, so here it is with its own proof.
tok.hlearns the o200k Split-regex family (case-aware letter runs, attached contractions,\p{N}{1,3}, the[\r\n/]*punctuation tail), selected per-tokenizer by detecting\p{Lu}in the tokenizer.json's own Split pattern. cl100k tokenizers can never take the new branch, so glm's path is unchanged by construction. New Unicode range tables live intok_unicode_o200k.h(Lu+Lt and Lm+Lo+M, generated from unicodedata).Verification, all runnable here:
tests/test_tok_o200k(added toTEST_BINS): a synthetic byte-level BPE (tests/tok_o200k_tiny.json, ~4 KB, committed) carries the o200k regex; expected ids were generated by HFtokenizersagainst that same file. 40/40 encode, 40/40 round-trip — case transitions, contractions, digit groups, slash tails, whitespace branches, CJK/Greek/Cyrillic, added-token atomicity.make check: 70/70. 0 warnings.tokenizersincluding 300 fuzz cases — not CI'd here since that file isn't committable, but reproducible withtests/test_tok.)If this lands, #312 rebases to purely additive files (new engine + new backend + new tools), per your comment there.
🤖 Generated with Claude Code
https://claude.ai/code/session_016KLDCYJSyHxd39ChTr4T27