Added new memory efficient conversion script for hf to ggml format, tested on bloomz 176B + Added token conversion script to convert from tokenizer.json format to tokenizer.model#867

Closed

akumaburn wants to merge 4 commits into

ggml-org:masterfrom

akumaburn:feature/tokenconvert

Commits on Apr 9, 2023

Added token conversion script to convert from tokenizer.json format to tokenizer.model format, tested with bigscience models
aeslampanah
committed

Commits on Apr 10, 2023

Add helper script to convert hf (pytorch) models into ggml format
aeslampanah
committed

Commits on Apr 13, 2023

Updated tokenconvert.py script to add support for SentencePiece and WordPiece tokenizers, updated arguments
aeslampanah
committed
Add sentencepiece processor
aeslampanah
committed