Question about training script

[cjfcsjt]

Hi, thanks for your great work!

May I ask if there is any plan or timeline to release the training script? Or could you please provide some guidelines on how to implement the training process?

According to the paper, I understand that in the sft dataset, each query contains six reasoning paths (with at least one correct path) concatenated with the final summary. I’m wondering whether the Sequence-Aware Positional Embedding, attention mask and positional ID parts need to be modified in the LLaMA-Factory source code during training?

[LeslieKid]

May I ask if there is any plan or timeline to release the training script?

We would release the training script and dataset in this month.

I’m wondering whether the Sequence-Aware Positional Embedding, attention mask and positional ID parts need to be modified in the LLaMA-Factory source code during training?

These parts should be customized for ParaThinker. But we do not modify the source code in LLaMA-Factory. Specifically, we

• Introduce sequence-aware positional embedding in transformers lib, which would be invoked in LLaMA-Factory.
• Customize a DataCurator class for specific attention mask and positional ID based on DataCollatorForSeq2Seq provided by LLaMA-Factory.