Tokenization for the large model

[ccharest93]

1. The config for large model specifies a vocab size of 51200, is there a separate tokenizer file for it? Weirdly vocab falls back down to 32 for xlarge which makes me think typo?
2. The tokenizer file specifies a vocab_size of 30, while the config for base and small specifies 32. Is this rounding to a power of two for efficiency?

[Lsz-20]

1. The config for large model specifies a vocab size of 51200, is there a separate tokenizer file for it? Weirdly vocab falls back down to 32 for xlarge which makes me think typo?
2. The tokenizer file specifies a vocab_size of 30, while the config for base and small specifies 32. Is this rounding to a power of two for efficiency?

Hello,have you solved this problem? I think it can embed longer sequence, but i found after the channel after model is 51200?

[zj-liu]

I think it is a typo. If you extract the parameters corresponding to the first 32 dims of the nn.embedding and lm_head in the model, it gives the same result. You can try the model here (https://huggingface.co/IDEA-XL/progen2-large).