-
Notifications
You must be signed in to change notification settings - Fork 3
InferenceEngines
Pantella has abstracted various methods of interfacing with LLMs as "Inference Engines". Inference Engines are created to add support to Pantella for different ways of running LLMs. The Inference Engine class is a base class that is inherited by the specific Inference Engine classes. The Inference Engine class contains the basic methods and properties that are required for all Inference Engines. The specific Inference Engine classes contain the methods and properties that are specific to the LLM that they are interfacing with. The Inference Engine class contains the following methods and properties:
The Inference Engine class is a base class that is inherited by the specific Inference Engine classes. The Inference Engine class contains the basic methods and properties that are required for all Inference Engines. The Inference Engine class has the following methods and properties:
-
__init__(self, config, conversation_manager): The constructor for the Inference Engine class. The constructor takes two parameters:configandconversation_manager. Theconfigparameter is a dictionary that contains the configuration settings for the Inference Engine. Theconversation_managerparameter is an instance of the Conversation Manager class that is used to manage the conversation. The constructor initializes the Inference Engine with the configuration settings and the Conversation Manager instance. -
generate_character(self, character_name, character_ref_id, character_base_id, character_in_game_race, character_in_game_gender, character_is_guard, character_is_ghost, in_game_voice_model, location): A method that generates a character using the LLM. The method takes the following parameters:character_name,character_ref_id,character_base_id,character_in_game_race,character_in_game_gender,character_is_guard,character_is_ghost,in_game_voice_model, andlocation. The method generates a character using the LLM and returns the generated character. -
get_cot_supported(self): A method that returns a boolean value indicating whether the Inference Engine supports the Cotatron model. -
cot_enabled: A property that returns a boolean value indicating whether the Cotatron model is enabled. -
end_of_sentence_chars: A property that returns a list of characters that are used to indicate the end of a sentence. -
stop: A property that returns a list of characters that are used to indicate the end of a sentence. -
replacements: A property that returns a list of replacement characters that are used to replace characters in the generated text. -
undo: A property that returns a list of characters that are used to indicate that the previous character should be removed. -
_prompt_style: A property that returns a dictionary that contains the prompt style settings for the Inference Engine. -
language: A property that returns the language of the Inference Engine. -
max_response_sentences: A property that returns the maximum number of sentences that the Inference Engine can generate in a response. -
behavior_style: A property that returns a dictionary that contains the behavior style settings for the Inference Engine. -
character_manager: A property that returns the Character Manager instance that is used to manage the characters in the conversation. -
game_interface: A property that returns the Game Interface instance that is used to interface with the game. -
maximum_local_tokens: A property that returns the maximum number of tokens that the Inference Engine can generate locally. -
player_name: A property that returns the name of the player. -
EOS_token: A property that returns the end of sentence token from the prompt style. -
BOS_token: A property that returns the beginning of sentence token from the prompt style. -
message_signifier: A property that returns the message signifier that is used to indicate that a message is from a player. -
message_format: A property that returns the message format that is used to format messages from players. -
message_separator: A property that returns the message separator that is used to separate messages from players. -
max_tokens: A property that returns the maximum number of tokens that the Inference Engine can generate. -
temperature: A property that returns the temperature that is used to control the randomness of the generated text. -
top_k: A property that returns the top_k value that is used to control the diversity of the generated text. -
top_p: A property that returns the top_p value that is used to control the diversity of the generated text. -
min_p: A property that returns the min_p value that is used to control the diversity of the generated text. -
logit_bias: A property that returns the logit_bias value that is used to control the diversity of the generated text. -
repeat_penalty: A property that returns the repeat_penalty value that is used to control the diversity of the generated text. -
tfs_z: A property that returns the tfs_z value that is used to control the diversity of the generated text. -
frequency_penalty: A property that returns the frequency_penalty value that is used to control the diversity of the generated text. -
presence_penalty: A property that returns the presence_penalty value that is used to control the diversity of the generated text. -
typical_p: A property that returns the typical_p value that is used to control the diversity of the generated text. -
mirostat_mode: A property that returns the mirostat_mode value that is used to control the diversity of the generated text. -
mirostat_eta: A property that returns the mirostat_eta value that is used to control the diversity of the generated text. -
mirostat_tau: A property that returns the mirostat_tau value that is used to control the diversity of the generated text. -
transformers_model_slug: A property that returns the slug of the Transformers model that is used by the Inference Engine. -
device_map: A property that returns the device map that is used to map devices to the Transformers model. -
trust_remote_code: A property that returns a boolean value indicating whether the Inference Engine trusts remote code to run when loading a model. -
load_in_8bit: A property that returns a boolean value indicating whether the Inference Engine loads the model in 8-bit mode. -
messages: A property that returns a list of messages from the ongoing conversation. -
_prompt_style: A property that returns a dictionary that contains the prompt style settings for the Inference Engine. -
ocr_resolution: A property that returns the resolution of the OCR model that is used by the Inference Engine. -
image_resolution: A property that returns the resolution of the image that is used by the Inference Engine. -
get_ascii_block(self, paddle_result, img, ascii_representation_max_size = 128): A method that returns an ASCII representation of the OCR result. -
get_player_perspective(self, check_vision=False): A method that returns the player's perspective. -
get_game_window(self): A method that returns the game window. -
__str__(self): A method that returns a string representation of the Inference Engine. -
chatgpt_api(self, input_text, messages): A deprecated method that is used to interface with the ChatGPT API. Use create() instead. -
create(self, messages): A method that is used to generate a complete response to a conversation. The method takes a list of messages as a parameter and returns a response to the conversation. -
acreate(self, messages, message_prefix="", force_speaker=None): A method that is used to generate a streaming response to a conversation. The method takes a list of messages as a parameter and returns a response to the conversation. -
clean_sentence(self, sentence): A method that cleans a sentence by removing unwanted characters and formatting the sentence. -
get_messages(self): A method that returns the messages from the conversation, this includes all memories, the system prompt, any images, and the latest player message. -
get_context(self): A method that returns the context of the conversation. Reformats the messages gotten from get_messages() into the correct format for the LLM. -
generate_response(self, message_prefix="", force_speaker=None): A method that generates a response to a conversation using the acreate() method. The method takes two optional parameters:message_prefixandforce_speaker. Themessage_prefixparameter is a string that is added to the beginning of the response. Theforce_speakerparameter is a string that forces the response to be spoken by a specific character. The method returns a response to the conversation. -
format_content(self, chunk): A method that formats the content of a message chunk as it's streamed to the game. -
split_and_preverse_strings_on_end_of_sentence(self, sentence, next_sentence=""): A method that splits and preserves strings on the end of a sentence. -
process_response(self, sentence_queue, event, force_speaker=None): A method that processes the response to a conversation. The method takes three parameters:sentence_queue,event, andforce_speaker. Thesentence_queueparameter is a queue that contains the sentences to be spoken. Theeventparameter is an event that is used to synchronize the generation of the sentences. Theforce_speakerparameter is a string that forces the response to be spoken by a specific character. The method generates the audio for the response and adds it to the sentence queue. -
check_author(self, proposed_next_author, next_author, verified_author, possible_players, retries, bad_author_retries, system_loop): A method that checks the author of the next sentence. -
generate_voiceline(self, string, sentence_queue, event): A method that generates audio for a voiceline.
The constructor for the Inference Engine class. The constructor takes two parameters: config and conversation_manager. The config parameter is a dictionary that contains the configuration settings for the Inference Engine. The conversation_manager parameter is an instance of the Conversation Manager class that is used to manage the conversation. The constructor initializes the Inference Engine with the configuration settings and the Conversation Manager instance. All Inference Engine classes should inherit from the Inference Engine class and call the constructor of the base class in their constructor.
A method that generates a character using the LLM. The method takes the following parameters: character_name, character_ref_id, character_base_id, character_in_game_race, character_in_game_gender, character_is_guard, character_is_ghost, in_game_voice_model, and location. The method generates a character using the LLM and returns the generated character. Not all Inference Engines will support this method.
A method that returns a boolean value indicating whether the Inference Engine supports the Cotatron model. Not all Inference Engines will support this method.
A property that returns a boolean value indicating whether the Cotatron model is enabled. Not all Inference Engines will support this property.
A property that returns a list of characters that are used to indicate the end of a sentence.
A property that returns a list of characters that are used to indicate the end of a sentence.
A property that returns a list of replacement characters that are used to replace characters in the generated text.
A property that returns a list of characters that are used to indicate that the previous character should be removed.
A property that returns a dictionary that contains the prompt style settings for the Inference Engine.
A property that returns the language of the Inference Engine.
A property that returns the maximum number of sentences that the Inference Engine can generate in a response.
A property that returns a dictionary that contains the behavior style settings for the Inference Engine.
A property that returns the Character Manager instance that is used to manage the characters in the conversation.
A property that returns the Game Interface instance that is used to interface with the game.
A property that returns the maximum number of tokens that the Inference Engine can generate locally.
A property that returns the name of the player.
A property that returns the end of sentence token from the prompt style.
A property that returns the beginning of sentence token from the prompt style.
A property that returns the message signifier that is used to indicate that a message is from a player.
A property that returns the message format that is used to format messages from players.
A property that returns the message separator that is used to separate messages from players.
A property that returns the maximum number of tokens that the Inference Engine can generate.
A property that returns the temperature that is used to control the randomness of the generated text.
A property that returns the top_k value that is used to control the diversity of the generated text.
A property that returns the top_p value that is used to control the diversity of the generated text.
A property that returns the min_p value that is used to control the diversity of the generated text.
A property that returns the logit_bias value that is used to control the diversity of the generated text.
A property that returns the repeat_penalty value that is used to control the diversity of the generated text.
A property that returns the tfs_z value that is used to control the diversity of the generated text.
A property that returns the frequency_penalty value that is used to control the diversity of the generated text.
A property that returns the presence_penalty value that is used to control the diversity of the generated text.
A property that returns the typical_p value that is used to control the diversity of the generated text.
A property that returns the mirostat_mode value that is used to control the diversity of the generated text.
A property that returns the mirostat_eta value that is used to control the diversity of the generated text.
A property that returns the mirostat_tau value that is used to control the diversity of the generated text.
A property that returns the slug of the Transformers model that is used by the Inference Engine.
A property that returns the device map that is used to map devices to the Transformers model.
A property that returns a boolean value indicating whether the Inference Engine trusts remote code to run when loading a model.
A property that returns a boolean value indicating whether the Inference Engine loads the model in 8-bit mode.
A property that returns a list of messages from the ongoing conversation.
A property that returns a dictionary that contains the prompt style settings for the Inference Engine.
A property that returns the resolution of the OCR model that is used by the Inference Engine.
A property that returns the resolution of the image that is used by the Inference Engine.
A method that returns an ASCII representation of the OCR result.
A method that returns the player's perspective.
A method that returns the game window.
A method that returns a string representation of the Inference Engine.
A deprecated method that is used to interface with the ChatGPT API. Use create() instead.
A method that is used to generate a complete response to a conversation. The method takes a list of messages as a parameter and returns a response to the conversation.
A method that is used to generate a streaming response to a conversation. The method takes a list of messages as a parameter and returns a response to the conversation.
A method that cleans a sentence by removing unwanted characters and formatting the sentence.
A method that returns the messages from the conversation, this includes all memories, the system prompt, any images, and the latest player message.
A method that returns the context of the conversation. Reformats the messages gotten from get_messages() into the correct format for the LLM.
A method that generates a response to a conversation using the acreate() method. The method takes two optional parameters: message_prefix and force_speaker. The message_prefix parameter is a string that is added to the beginning of the response. The force_speaker parameter is a string that forces the response to be spoken by a specific character. The method returns a response to the conversation.
A method that formats the content of a message chunk as it's streamed to the game.
A method that splits and preserves strings on the end of a sentence.
A method that processes the response to a conversation. The method takes three parameters: sentence_queue, event, and force_speaker. The sentence_queue parameter is a queue that contains the sentences to be spoken. The event parameter is an event that is used to synchronize the generation of the sentences. The force_speaker parameter is a string that forces the response to be spoken by a specific character. The method generates the audio for the response and adds it to the sentence queue. This is the most complicated and most important method in the Inference Engine class, this handles how each token is processed and how the response is generated. This is also where the Narrator/Speaker logic is implemented.
check_author(self, proposed_next_author, next_author, verified_author, possible_players, retries, bad_author_retries, system_loop) Method
A method that checks the author of the next sentence.
A method that generates audio for a voiceline.
print("importing example.py")
from src.logging import logging, time
import src.utils as utils
from src.inference_engines.base_llm import base_LLM, TestCoT, get_schema_description
logging.info("imported example.py")
inference_engine_name = "example"
class LLM(base_LLM):
def __init__(self, config, conversation_manager):
super().__init__(config, conversation_manager)
logging.info(f"Initializing {inference_engine_name} Inference Engine")
# Add any additional initialization code here
logging.info(f"Initialized {inference_engine_name} Inference Engine")
def generate_character(self, character_name, **kwargs):
# Add character generation code here if it is supported
return None
def create(self, messages):
# Add code to generate a response to a conversation here
return None
def acreate(self, messages, message_prefix="", force_speaker=None):
# Add code to generate a streaming response to a conversation here
return None