-
Notifications
You must be signed in to change notification settings - Fork 3
Tokenizers
Tokenizers are one of the more unique modules in Pantella, as they are defined in a few ways. Most often they are created in the ./src/tokenizers/ directory, but they can also be created in an Inference Engine's python script, like ./src/inference_engines/llama_cpp_python.py has it's own Tokenizer class. Tokenizer classes created with an inference engine are typically going to be better than the generic ones as they're most likely tied into the actual tokenizer the model is using under the hood and will allow for accurate token counts, but estimates are also usable if that's all you can get. The Tokenizer class is used to tokenize text input from the player. Token counts are used to estimate/calculate the length of a prompt before it's sent to the LLM. This is used to allocate budgets to which memories are available in context, which recent messages are available in context, etc. The Tokenizer class has the following methods:
-
__init__(self,conversation_manager): Initializes the Tokenizer. -
new_message(self, content, role, name=None): Parses a string into a message format with the name of the speaker. -
start_message(self, role="", name=None): Returns the start of a message with the name of the speaker. -
end_message(self, role="", name=None): Returns the end of a message with the name of the speaker. -
get_string_from_messages(self, messages): Returns a formatted string from a list of messages. -
num_tokens_from_messages(self, messages): Returns the number of tokens used by a list of messages. -
get_token_count(self, string): Returns the number of tokens in a string.
Initializes the Tokenizer. It loads the config and sets the tokenizer slug. The conversation_manager parameter is the conversation manager that the Tokenizer is associated with.
Parses a string into a message format with the name of the speaker. The content parameter is the content of the message. The role parameter is the role of the speaker. The name parameter is the name of the speaker. The method returns a string that contains the parsed message.
Returns the start of a message with the name of the speaker. The role parameter is the role of the speaker. The name parameter is the name of the speaker. The method returns a string that contains the start of the message.
Returns the end of a message with the name of the speaker. The role parameter is the role of the speaker. The name parameter is the name of the speaker. The method returns a string that contains the end of the message.
Returns a formatted string from a list of messages. The messages parameter is a list of messages. The method returns a string that contains the formatted messages.
Returns the number of tokens used by a list of messages. The messages parameter is a list of messages. The method returns the number of tokens used by the messages.
Returns the number of tokens in a string. The string parameter is the string that is being evaluated. The method returns the number of tokens in the string.
When creating a new Tokenizer, you should inherit from the base Tokenizer class. This ensures that the Tokenizer has access to the base Tokenizer class's methods and properties. To inherit from the base Tokenizer class, you should
print("Importing word_count_tokenizer.py...")
from src.logging import logging
import src.utils as utils
import src.tokenizers.base_tokenizer as tokenizer
logging.info("Imported required libraries in word_count_tokenizer.py")
tokenizer_slug = "word_count"
class Tokenizer(tokenizer.base_Tokenizer): # Tokenizes(only availble for counting the tokens in a string presently for local_models), and parses and formats messages for use with the language model
def __init__(self,conversation_manager):
super().__init__(conversation_manager)
self.tokenizer_slug = tokenizer_slug
@utils.time_it
def get_token_count(self, string):
"""Returns the number of tokens in the string"""
tokens = string.split(" ")
num_tokens = len(tokens)
return num_tokens