If we set the init checkpoint for a model to e.g. https://huggingface.co/PsycheFoundation/moe-10b-a1b-init , but then set the HF repo for the checkpointer nodes to a blank repo e.g. https://huggingface.co/NousResearch/moe-10b-a1b-8k-wsd-lr3e4-1t/tree/main , only the safetensor files are uploaded. we should, in checkpointing, also upload any files from the original repo that are not in this repo . in the example urls here, it should be added_tokens.json, config.json, merges.txt, special_tokens_map.json, tokenizer.json, tokenizer_config.json, and vocab.json .
If we set the init checkpoint for a model to e.g. https://huggingface.co/PsycheFoundation/moe-10b-a1b-init , but then set the HF repo for the checkpointer nodes to a blank repo e.g. https://huggingface.co/NousResearch/moe-10b-a1b-8k-wsd-lr3e4-1t/tree/main , only the safetensor files are uploaded. we should, in checkpointing, also upload any files from the original repo that are not in this repo . in the example urls here, it should be added_tokens.json, config.json, merges.txt, special_tokens_map.json, tokenizer.json, tokenizer_config.json, and vocab.json .