Training Cost

[Yeez-lee]

Dear Authors,

Thanks for an interesting idea and open-sourced codes. I wonder how many wallclock-wise hours can be saved when using all_4b_faster_eval to skip some evaluations on 4 H200 NVIDIA GPUs.

[Blinorot]

Hello!

Thank you for your interest in our work! If I remember correctly, in our setup, evaluation on the full validation set took roughly 10 hours on 4×H200 GPUs, while all_4b_faster_eval reduced this to about 1–2 hours, mainly because it skips some of the validation datasets.

Even with the reduced set, faster_eval still provides a useful signal about whether the model is improving on speech/sound/music across epochs.

In our experience, during the first pass over the full validation corpora the validation loss steadily decreases when evaluated every 20k steps, so during the first epoch it can make sense to run validation less frequently.

Another possible speed-up is skipping sampling example responses (via sample_generations=False) and saving checkpoints more frequently, which can make it easier to resume runs in time-constrained setups (e.g., SLURM jobs). Note that checkpoint saving and evaluation frequencies can be controlled independently via eval_steps and save_steps.

Please note that the exact wall-clock numbers may vary depending on the setup (CPU, storage speed such as HDD vs SSD, etc.).

The overall training time is provided in the .sh scripts for ALARM-CA/P. The numbers are approximate, but in general training the multi-encoder variant on the full corpora takes about 100–120 hours per epoch on our setup.