[no-merge] add fc yaml#4710
Conversation
| sharding_parallel_size: 4 | ||
| pp_delay_scale_loss: true | ||
| use_expert_parallel: true | ||
| expert_model_parallel_size: 4 |
There was a problem hiding this comment.
这个 2nodes 配置按同目录 FT 配置的 2 节点/8 卡约定运行时会在训练参数初始化阶段直接失败。TrainingArguments._post_init_parallel_degree() 会计算 data_parallel_size = world_size / (sharding_parallel_size * tensor_model_parallel_size * sep_parallel_size * pipeline_model_parallel_size);16 卡下这里是 16/(4*1*1*2)=2,同时 expert_model_parallel_size=4 又得到 moe_sharding_parallel_size=16/(2*4)=2,随后代码明确拒绝 moe_sharding_parallel_size > 1 && data_parallel_size > 1 以及 data_parallel_size > 1 && expert_model_parallel_size > 1 的组合,训练不会启动。若这个文件仍是 2 节点 8 卡场景,请保持 sharding/EP 度数覆盖完整每个 PP stage;若目标其实是 2 节点 4 卡,也需要在文件名或注释里标清并配套启动脚本固定 world_size=8。
| sharding_parallel_size: 4 | |
| pp_delay_scale_loss: true | |
| use_expert_parallel: true | |
| expert_model_parallel_size: 4 | |
| sharding_parallel_size: 8 | |
| pp_delay_scale_loss: true | |
| use_expert_parallel: true | |
| expert_model_parallel_size: 8 |
| sharding_parallel_size: 4 | ||
| pp_delay_scale_loss: true | ||
| use_expert_parallel: true | ||
| expert_model_parallel_size: 4 |
There was a problem hiding this comment.
这个 3nodes 配置也会在常规 3 节点/8 卡的 FT 运行里被并行度校验拒绝。24 卡下当前值会得到 data_parallel_size = 24/(4*1*1*3)=2,同时 moe_sharding_parallel_size = 24/(3*4)=2;TrainingArguments._post_init_parallel_degree() 不支持 EP 和数据并行同时大于 1,也不支持 moe_sharding_parallel_size > 1 时再有 data_parallel_size > 1,因此训练初始化会抛错。若该文件表示 3 节点 8 卡,请改回覆盖完整 PP stage 的 sharding/EP 度数;若实际只打算 4 卡/节点运行,请把这个 rank 约束写进配置说明/文件名,避免按 3nodes 直接使用时失败。
| sharding_parallel_size: 4 | |
| pp_delay_scale_loss: true | |
| use_expert_parallel: true | |
| expert_model_parallel_size: 4 | |
| sharding_parallel_size: 8 | |
| pp_delay_scale_loss: true | |
| use_expert_parallel: true | |
| expert_model_parallel_size: 8 |
Before submitting
testsfolder. If there are codecov issues, please add tests cases first.PR types
Others
PR changes
Others
Description
add fc yaml