perf(bpe): pack a short word into its key instead of hashing it - #2315

perf(bpe): pack a short word into its key instead of hashing it#2315

ArthurZucker merged 1 commit into

perf/bpe-hash-word-oncehuggingface/tokenizers:perf/bpe-hash-word-oncefrom

perf/bpe-pack-short-keyshuggingface/tokenizers:perf/bpe-pack-short-keysCopy head branch name to clipboard