deps: update tiktoken-rs requirement from 0.5.8 to 0.12.0 in /crablet

[dependabot]

Updates the requirements on tiktoken-rs to permit the latest version.

Release notes

Sourced from tiktoken-rs's releases.

v0.12.0

Summary

This release backports OpenAI tiktoken 0.13.0 into tiktoken-rs. The main reason to upgrade is better alignment with upstream tokenization behavior, especially the upstream Rust core changes for large BPE pieces and error-aware encoding.

For most users who call the high-level model/token counting helpers, this should behave the same aside from the new Rust compiler requirement. Users who call lower-level CoreBPE encoding methods directly should review the breaking changes below.

What Changed

• Backported the vendored OpenAI tiktoken Rust core from 0.9.0 to 0.13.0.
• Added the upstream large-piece BPE merge path. Functionally, this improves behavior for very large or repetitive inputs that previously stressed the merge algorithm.
• Changed CoreBPE::encode to return Result<(Vec<Rank>, usize), EncodeError>, matching upstream. Regex/tokenization failures can now be reported instead of being hidden behind infallible APIs.
• Updated encode_as and count to return Result because they call encode.
• Re-exported EncodeError so callers can handle encode failures directly.
• Aligned the vendored core with Rust 2024 and raised the crate MSRV to Rust 1.85.
• Synced model-to-tokenizer mappings with upstream tiktoken 0.13.0 while keeping local extra prefixes isolated.
• Hardened asset downloads with SHA-256 checks and a repo-root-aware asset path.

Breaking Changes

If your code calls CoreBPE::encode, unwrap or propagate the result before using the tokens:

let allowed = bpe.special_tokens();
let (tokens, last_piece_token_len) = bpe.encode("hello <|endoftext|>", &allowed)?;

The generic helpers changed similarly:

let (tokens, last_piece_token_len) = bpe.encode_as::<usize>(text, &allowed)?;
let token_count = bpe.count(text, &allowed)?;

encode_ordinary, encode_ordinary_as, encode_with_special_tokens, and count_ordinary remain infallible.

Projects must now build with Rust 1.85 or newer.

Practical Impact

• Applications processing long repeated text should see more robust tokenization behavior.
• Code that only uses helpers like get_chat_completion_max_tokens, get_text_completion_max_tokens, bpe_for_model, or singleton tokenizer constructors should not need call-site changes.
• Code using low-level CoreBPE::encode, encode_as, or count needs a small migration to handle Result.

Links

• PR: zurawiki/tiktoken-rs#164
• Upstream tiktoken 0.13.0: https://github.com/openai/tiktoken/releases/tag/0.13.0
• Full changelog: zurawiki/tiktoken-rs@v0.11.0...v0.12.0

Commits

• 32de8dc Bump version minor to "0.12.0"
• ac7e2e8 Backport tiktoken 0.13.0 (#164)
• b115197 Return Option from get_context_size, improve error messages, bump to 0.11.0 (...
• 5561968 Add gpt-5.2/5.3/codex model support (#159)
• 7555342 Add generic token type APIs (FromRank trait) (#157)
• 415f11e Use GitHub auto-generated release notes (#158)
• 5f47553 Add gpt-5.4 support, fix o1-mini/chatgpt-4o/gpt-4.5 context sizes, DRY ft: pr...
• 4194603 Auto-label Renovate PRs as dependencies
• 1a9799a Release v0.10.0 (#155)
• 5ed40bb Add count-only token counting APIs (#153)
• Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

---

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot show <dependency name> ignore conditions will show all of the ignore conditions of the specified dependency
• @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

[dependabot]

Labels

The following labels could not be found: dependencies, rust. Please create them before Dependabot can add them to a pull request.

Please fix the above issues or remove invalid values from dependabot.yml.