Comparsion with Muon optimizer

[zhuhanqing]

Hi authors,

Thank you for your great work. I just have a question about the performance between Muon and Mars. Is Mars strictly better than Muon?

[lauyikfung]

Thank you for you attention to our work. We implemented several experiences, including CV tasks on CIFAR10/CIFAR100 datasets and GPT-2 series models (with hyper-parameter search), where MARS outperforms muon on all these tasks. We will do more experiences with MARS and baseline optimizers (including AdamW and Muon) as well as provide the theoretical analysis on the convergence of MARS, and update them to this repo :)

[zhuhanqing]

Thank you so much for your response. I am wondering do you have any suggestions on tuning hyper-parameter for Mars to ensure a strong performance.

Currently, on LLaMA-130M with seq 256, I sweep the lr in 0.0025 0.005 0.0075 0.01, with beta1 0.9 and find the lr 0.0025 is best with perplexity 23.35. Do you suggest tuning some other hyperparameter? Or do you have to plan to include performance on the LLaMA series?

[angelahzyuan]

Thank you for your interest in our work. The LLaMA series is next on our to-do list.

[sozforex]

Would be interesting to see if MARS can outperform Muon on the wall-time to 94% acc on cifar10:
https://github.com/KellerJordan/cifar10-airbench