Add @stdlib/tokenizer module: BPE tokenizer for tiktoken-compatible token counting#441
Open
nbeerbower wants to merge 1 commit into
Open
Add @stdlib/tokenizer module: BPE tokenizer for tiktoken-compatible token counting#441nbeerbower wants to merge 1 commit into
nbeerbower wants to merge 1 commit into
Conversation
…oken counting
Implements a full BPE (Byte Pair Encoding) tokenizer as a new stdlib module,
enabling Hemlock programs to count tokens and stay under LLM context limits.
Core features:
- Load tiktoken-format vocabulary files (base64 token bytes + rank)
- Encode text to token IDs via standard BPE merge algorithm
- Decode token IDs back to text
- Count tokens efficiently
- Add special tokens at runtime
- Full interpreter + compiler parity
Implementation:
- C builtins for interpreter (src/backends/interpreter/builtins/tokenizer.c)
- Runtime library for compiler (runtime/src/builtins_tokenizer.c)
- Hemlock stdlib wrapper with Tokenizer object API (stdlib/tokenizer.hml)
- Compiler codegen for all 7 builtin functions
- Registered in builtins_registry.h, internal.h, registration.c
API:
import { Tokenizer } from "@stdlib/tokenizer";
let tok = Tokenizer("vocab.bpe");
let tokens = tok.encode("Hello, world!");
let count = tok.count("Hello, world!");
let text = tok.decode(tokens);
tok.free();
https://claude.ai/code/session_019TgFT84gP7VHFPLPThQRuG
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements a full BPE (Byte Pair Encoding) tokenizer as a new stdlib module,
enabling Hemlock programs to count tokens and stay under LLM context limits.
Core features:
Implementation:
API:
import { Tokenizer } from "@stdlib/tokenizer";
let tok = Tokenizer("vocab.bpe");
let tokens = tok.encode("Hello, world!");
let count = tok.count("Hello, world!");
let text = tok.decode(tokens);
tok.free();
https://claude.ai/code/session_019TgFT84gP7VHFPLPThQRuG