Question regarding the Yume 1.5 paper (about the foundation model and distillation method)

[LeeKeyu]

Hi authors, thanks for your inspiring work!

After reading through your YUME1.5 paper, I have a question about the foundation model. You mentioned "we first initialize the generator Gθ, fake model Gs, and real model Gt with weights from a foundation model [15]. " Do you mean the foundation model is already an autoregressive diffusion model and DMD distillation is only used to reduce the number of denoising steps? (as also indicated by Fig. 4, to my understanding).
Then in the Experiments part, you mentioned "We utilized the Wan2.2-5B as the pre-trained model. " This makes me a little confused. Do you mean in the foundation model, 1) an autoregressive generation model architecture like the in the self-forcing paper is used, but the base model is changed from their Wan 1.3B t2v to Wan 2.2 5B, and then, 2) reduce the number of denoising steps by DMD distillation, without changing the autoregressive model architecture? Or the bidirectional Wan 2.2 5B is trained as the foundation model and then distilled to an autoregressive model to take into account history information?

Thanks so much for your help in advance!

[stdstu12]

We conduct a two-stage training pipeline. In Stage 1, the model undergoes T2V and I2V training through the TSCM module (Sec. 4.2), establishing Yume1.5 as an autoregressive diffusion model. In Stage 2, we eliminate the KV Cache from the Self-Forcing mechanism and use the TSCM to compress historical frames for autoregressive inference. This method proves efficient: it enables extrapolation to half a minute with only 64 frames of training, and eliminates the need for ODE initialization.

[LeeKeyu]

We conduct a two-stage training pipeline. In Stage 1, the model undergoes T2V and I2V training through the TSCM module (Sec. 4.2), establishing Yume1.5 as an autoregressive diffusion model. In Stage 2, we eliminate the KV Cache from the Self-Forcing mechanism and use the TSCM to compress historical frames for autoregressive inference. This method proves efficient: it enables extrapolation to half a minute with only 64 frames of training, and eliminates the need for ODE initialization.

@stdstu12 Hi thanks so much for your reply! Do you mean both Stage 1 and 2 used the same autoregressive model architecture with TSCM? Are there any differences in model architecture between the two stages? Also, as Wan 2.2 5B is a bidirectional diffusion model, did you add the TSCM modules trained from scratch while also finetuning the original pre-trained parameters of Wan in Stage 1?
May I ask when will you release the full training code for the 5B model? Thanks a lot!

[stdstu12]

Yes, we initialized with Wan5B, and then used the same autoregressive model architecture and TSCM for all stages. I provided training code in Issue #16 that should theoretically work. If you encounter any issues during training, please let me know, and I will try to fix them.