RT, **Tokenizer::encode** function is very poor performance ! some places has need improve or move https://github.com/NVIDIA/TensorRT-Edge-LLM/blob/release/0.8.0/cpp/tokenizer/tokenizer.cpp#L489 https://github.com/NVIDIA/TensorRT-Edge-LLM/blob/release/0.8.0/cpp/tokenizer/tokenizer.cpp#L492 https://github.com/NVIDIA/TensorRT-Edge-LLM/blob/release/0.8.0/cpp/tokenizer/tokenizer.cpp#L452 and **partitionSpecialTokens** and **bytePairEncode** function can ref llama.cpp to restruct.
RT, Tokenizer::encode function is very poor performance !
some places has need improve or move
https://github.com/NVIDIA/TensorRT-Edge-LLM/blob/release/0.8.0/cpp/tokenizer/tokenizer.cpp#L489
https://github.com/NVIDIA/TensorRT-Edge-LLM/blob/release/0.8.0/cpp/tokenizer/tokenizer.cpp#L492
https://github.com/NVIDIA/TensorRT-Edge-LLM/blob/release/0.8.0/cpp/tokenizer/tokenizer.cpp#L452
and partitionSpecialTokens and bytePairEncode function can ref llama.cpp to restruct.