Tokenizer ignoring commas?

[Nguyen-Bang]

Hello Hustcw,

Thank you so much for doing this project. I was thinking of incorporating your tokenizer into my project. I noticed that you didn't include the "," in your vocab so its encoding it as "UNK". Is there a reason why you decide to skip the punctuation? Was it not important?
Sorry, if this is a dumb question. I just started working on this project.

Thank you so much

[Nguyen-Bang]

I guess you can ignore it since an instruction is will only consist of 3 components? and by doing so you can reduce the amount of input tokens?

[Nguyen-Bang]

I just took a closer look at all the vocab you have. It is missing the following tokens (esp, rip, ymm0, zmm0, xmm0, je, jne, ymmword) . Is there a reason for that.

[Dreamerxic]

You can see in the vocab.txt that some tokens start from "##". Those are "subword" marks. A token starting from "##" means it must follow another token.
esp -> e ##sp
rip -> r ##ip