Questions about method, dataset, and training cost

[LeeKeyu]

Hi authors,

Thanks for sharing your inspiring work! After carefully reading through your paper, I have some questions regarding the method and dataset:

1. In the two-stage training, you mentioned that in stage 1 only the VGGT part is trained with video generation part from Wan frozen, and camera control is added in Stage 2. Why not separately train the camera control modules before stage 1?
2. Could you elaborate more on the dataset processing method used in this work, i.e., the reconstruction method to "generate multiview consistent depth maps using a reconstruction-based pipeline"?
3. The minimum GPU hardware requirement to train and test this method.
4. Is the FantasyWorld-1.0 on the leaderboard assosiated with the Wan 2.2 version in this repository?

Thanks a lot for your help in advance!

[daiyixiang666]

Hi, thanks for your interest in our work.

1. Indeed, a more appropriate approach would be to start from a fully pre-trained camera-control module. However, we could not find a suitable pre-trained one, so we trained it ourselves. Training a camera-control module in Wan is actually more time-consuming than Stage 1, so we integrated this training into Stage 2.
2. This part can follow the pipeline described in Cut3R.
3. The training requirements are mentioned in the paper. As for evaluation on WorldScore, it is quite GPU-consuming: it includes around 1000 scenes, so the inference cost depends on how many GPUs you have available.
4. I’ll ask my teammate to answer this question.

If you have more questions, feel free to ask me!

[zhongyingji]

Hello Yixiang,

Thanks for your great work! I have a few questions about the training process and model details.

1. The entire training process is divided into two stages. In the first stage, only VGGT is trained, while in the second stage, VGGT and DiT are trained together. During the first stage of training, will cross-attention with camera embedding be added to the first 16 shared blocks? My impression is that Wan2.1 only supports single-image-to-video generation. If camera embedding is not added, the embeddings after 16 layers should be uncontrollable. Wouldn’t directly applying geometric supervision cause issues?

2. In stage 2, “Camera parameters are embedded with a pose encoder, xxx, applied to the first 24 of 40 blocks.”
   a. From my understanding, does this mean that cross-attention with camera embedding is only added in stage 2?
   b. Is camera embedding applied to the first 24 blocks or the first 16 blocks? Since the first 16 blocks are shared, why is camera embedding specifically applied to the first 24 blocks?
   c. In your code for Wan2.1, I noticed that camera embedding is actually added to all 40 blocks. So, in practice, during stage 2 training, which layers receive camera embedding?

Thank you. : )