Reproducing CIFAR10 result from paper: Improved Techniques For Consistency Training

[noamelata]

Hi!
Has anyone managed to reproduce the results from "Improved Techniques For Consistency Training" on the CIFAR10 dataset?

Thank you for the great repository!

[Kinyugo]

Hello!

I am not sure about this. So far I haven't carried out large experiments with this.

Thank you for the kind sentiments.

[thwgithub]

@Kinyugo @noamelata
Hi，everyone. I have implemented the experiment on butterflies dataset(len=1000), which the result trained 10000 steps(max_step) is better, FID5=0.2028 with five steps sampling, and FID1 =0.2165 with one step sampling. However, on cifar10, I didn't get good result as I expected. I set : (1) N(k)=11, max_steps=400000, optimizer=RAdam(better than Adam), batchsize=32 , result: FID10(10 step sampling)=74.3594, FID5=61.3095, FID2=127.9843, FID1=285.8623; (2) N(k)=1280, max_steps=400000, optimizer=RAdam , batchsize=32, result: FID10=64.5555,FID5=56.4666, FID2=68.3255, FID1=243.7875; (3)N(k)=1280, max_steps=400000, optimizer=RAdam , batchsize=64, result: FID10=30.2732, FID5=35.0473, FID2,=62.3042 ,FID1=238.0072. If I set the batchsize=128, The results were also poor.

[thwgithub]

I suspect there is something wrong with the structure of Unet, but i am not sure about it.

[Kinyugo]

Hello,
Kindly try the experiments with the unet from the paper. This was just a random unet. If you manage to get it working kindly share your findings.

[thwgithub]

@Kinyugo
Hello, thank you for your instant reply. I found a error in your codes as follows ：
in the timesteps_schedule function,
def timesteps_schedule(....):
num_timesteps = final_timesteps2 - initial_timesteps2
# the final_timesteps above should be "final_timesteps+1",

cf. page 26 in the Appendices of the paper ''Consistency Models" .

num_timesteps = current_training_step * num_timesteps / total_training_steps
num_timesteps = math.ceil(math.sqrt(num_timesteps + initial_timesteps**2) - 1)

return num_timesteps + 1

[Kinyugo]

Thanks for the nice find. I'll correct asap.

[thwgithub]

@Kinyugo However, this trivial error should not obscure the concision aesthetics of your code, this is an excellent work. As far as i know , your code is first one performed the "improved consistency models" in github.

[Kinyugo]

Thank you for your kind sentiments.

Regarding the model, I agree with you that it might be sub-optimal. A recent paper "Analyzing and Improving the Training Dynamics of Diffusion Models" could be of assistance, though it's based on the EDM paper. I do plan to experiment with the architecture there but I am currently held up. You could also check issue #7 that proposes rescaling of sigmas. I think with some of those changes you might get better results.

Thanks for taking the time to share you findings with me.

[thwgithub]

@Kinyugo Thanks for sharing your findings with me. I have not pay attention to the paper "[Analyzing and Improving the Training Dynamics of Diffusion Models]". Regarding the issue #7 , it is worth to try it. But i want to change firstly your unet structure with ncsn++ ,and then check the result on cifar10. To facilitate communication, we send emails(hongwei_tan@foxmail.com) each other.

[Kinyugo]

Awesome. Ping me incase of any questions.

If you replicate the ncsn++ network and get good results consider contributing to the repo.

[aiihn]

@thwgithub Hello! I was wondering if you have had the opportunity to reproduce the FID on CIFAR10 using the U-Net described in the "Consistency Models" paper. If so, may I kindly inquire about the results you obtained? Thanks in advance!

[thwgithub]

@aiihn sure.

[aiihn]

@thwgithub Thanks! What FID results have you obtained on cifar10?

[thwgithub]

@Kinyugo Hello, I have successfully incorporated the ncsn++ network(https://github.com/openai/consistency_models_cifar10/blob/main/jcm/models/ncsnpp.py) into your consistency model. Unfortunately, I did not still achieve good results. At the same time, I looked repeatedly your codes over, I assured it was no problem. Now, I am quite confused about it. Can you help me to check my codes?