Background
SentencePieceTokenizer.CreateFromTokenizerJson (added in #7625) loads a Hugging Face tokenizer.json for Unigram models. It reads model, normalizer, pre_tokenizer, added_tokens, and post_processor, but does not read the decoder section.
This was called out in review of #7625: it is fine for the standard HF Unigram/Metaspace configurations the current tests cover (decoding is driven by the SentencePiece model semantics), but it means non-standard decoder chains in a tokenizer.json are ignored rather than honored.
Ask
Support reading the tokenizer.json decoder section when a real model requires a decoder chain that differs from the implied SentencePiece/Metaspace decoding, e.g. Sequence, Replace, Strip, ByteFallback, Metaspace, Fuse decoders. Until then, the loader silently relies on the model''s built-in decode behavior.
Notes
Filed as a follow-up to review feedback on #7625.
Background
SentencePieceTokenizer.CreateFromTokenizerJson(added in #7625) loads a Hugging Facetokenizer.jsonfor Unigram models. It readsmodel,normalizer,pre_tokenizer,added_tokens, andpost_processor, but does not read thedecodersection.This was called out in review of #7625: it is fine for the standard HF Unigram/Metaspace configurations the current tests cover (decoding is driven by the SentencePiece model semantics), but it means non-standard
decoderchains in atokenizer.jsonare ignored rather than honored.Ask
Support reading the
tokenizer.jsondecodersection when a real model requires a decoder chain that differs from the implied SentencePiece/Metaspace decoding, e.g.Sequence,Replace,Strip,ByteFallback,Metaspace,Fusedecoders. Until then, the loader silently relies on the model''s built-in decode behavior.Notes
unk_id, and rejecting unrecognizedpre_tokenizertypes withNotSupportedException.Filed as a follow-up to review feedback on #7625.