LLM Tokenizers

Code by 🧑‍💻Dark Ngo.

Overviews

Welcome to LLM Tokenizers! This repository provides a tool to easily explore the world of large language models by entering your text, selecting your preferred LLM, and receiving a detailed tokenized output. Our system provides a list of tokens, token IDs, token count, and character count, all in an instant. Discover how language models process text with our user-friendly and intuitive tool.

Check out the live version at LLM Tokenizers.

Key Features

• Easy Text Tokenization: Enter your text and receive detailed tokenized output instantly.
• Multiple LLM Support: Select from various large language models to see how each processes the input text.
• Detailed Outputs: Get a comprehensive list of tokens, token IDs, token count, and character count.
• User-Friendly Interface: Our tool is designed to be intuitive and easy to use.
• API Access: For developers, API access is available. Visit our Swagger documentation to learn more.

Demo

If you like and find this repository useful, please consider giving me a ⭐.

Todos

• GPT
• Gemini
• Claude (1; 2)
• Claude (3; 3.5)
• LLama
• Mistral