tokenizer tokenizer

Constructors

tokenizer(std::filesystem::path path, bool input = false)

Reads saved vocab if input is true otherwise initializes new vocab.

---

Basic Info

A tokenizer using a greedy algorithm.

Instance Variables

Variable	Type
vocab	std::vector<std::string>
path	std::filesystem::path

---

Functions

• add_token()
• decode()
• encode()
• get_path()
• get_vocab()
• save_tokens()
• set_path()
• set_vocab()

---

Example

// Initializing a tokenizer
tokenizer t("./tokenstokenstokensssss.txt");