Help！！

[M0ck4ry]

Thank you very much for publishing your code. I have successfully reproduced the effect in the paper. But during testing, the loss will turn into nan around 80-100 rounds. May I ask why this is happening? How can I do better? I hope to receive your reply, thank you.

[PurmaVishnuVardhanReddy]

Thanks for reaching out. Could you elaborate on testing in 80-100 rounds ?

[M0ck4ry]

Thank you for your reply. I downloaded the Glas dataset as mentioned in the readme and trained it for data processing, pre training, and downstream training. In the 85th round of downstream training, there was a situation where the loss changed to 'nan'. Then, during testing, I inputted the 84th weight to the test, and the result was slightly lower than the indicators in the paper. Looking forward to hearing back from you again.

[PurmaVishnuVardhanReddy]

You can try the following:

• If there is log(x) present in the losses, try to add the stability factor i.e., log(x+1e-4)
• check for the gradient explode

[M0ck4ry]

Thank you again for your reply. I will try the solution you mentioned. In short, this is a great job that has provided me with a lot of rewards. Thank you again.