Code for YUME1.5

[wine-plum]

Great work! However, it seems that the current codebase is only compatible with YUME 1.0. Although the inference module provides a script for the 5B model, the code invoked the script still belongs to YUME 1.0 when actually running. For the training code, I haven't verified yet whether it supports YUME 1.5.

Therefore, I would like to ask: is there any plan to open-source the complete code specifically for YUME 1.5?

Additionally, I noticed that the YUME 1.0 code calls modules from the "wan" folder (wan-2.1), whereas YUME 1.5 should presumably call modules from the "wan23" folder (wan-2.2). Based on this, could switching to YUME 1.5 simply be achieved by modifying all existing references from "wan" to "wan23"? I am currently attempting this modification on my own, but I am not sure about its feasibility.

[flt-ime]

Hi @wine-plum, I'm very interested in using this repo. I noticed sample_5b.py does include some YUME 1.5-claimed changes, like text-controlled event generation with an extra prompt and the use of InternVL-2B-Instruct for refining event descriptions.
Could you clarify which parts of YUME 1.5 (e.g., TSCM or other components) are still not fully implemented in the current code? A confirmation would really help so I don't waste time hunting for missing code. Thanks!

[wine-plum]

First, I apologize for some of my incorrect statements above. To clarify, this repository does provide the inference script for YUME 1.5—the only issue is that there are some descriptive errors in the README file. Specifically, the inference script for the 5B model should be scripts/inference/sample_5b.sh, rather than scripts/inference/sample_tts.sh. At present, I have successfully run the T2V function of this script; however, due to GPU memory limitations, I have only managed to implement two rounds of autoregressive generation. As an additional note, the I2V module in the inference code leverages “InternVL3-2B-Instruct” for image processing, which needs to be downloaded separately. If possible, I suggest the authors add this clarification to the README file and revise the description of the inference script for YUME 1.5.

Second, regarding the training scripts, they do seem to still be targeted at YUME 1.0. However, I noticed that the author provided a training file in Issue #16, though I haven’t had time to examine it in detail yet. Next, I will attempt to reproduce the training process of YUME 1.5 with the help of these resources. That said, it would be greatly appreciated if the author could release the complete training workflow.

[wine-plum]

Hi @wine-plum, I'm very interested in using this repo. I noticed sample_5b.py does include some YUME 1.5-claimed changes, like text-controlled event generation with an extra prompt and the use of InternVL-2B-Instruct for refining event descriptions. Could you clarify which parts of YUME 1.5 (e.g., TSCM or other components) are still not fully implemented in the current code? A confirmation would really help so I don't waste time hunting for missing code. Thanks!

My apologies, I’ve only managed to get the inference code running successfully so far. As for the training-related content, I’m not sure about the specific details yet.

[stdstu12]

Apologies. As I mentioned earlier, I do not have access to any GPUs at the moment, so I cannot currently provide code that is guaranteed to run bug-free.

I have provided training code in Issue #16 that works in theory. If you encounter any issues during training, please let me know, and I will try to fix them.

Additionally, regarding the autoregressive generation stopping after just 2 steps—how much VRAM does your GPU have? I am investigating the issue, and a possible cause is that the current sampling code is running in FP32 inference. However, I was previously able to reach about 5 steps on the webapp_single_gpu.py using a 16GB GPU.

[stdstu12]

First, I apologize for some of my incorrect statements above. To clarify, this repository does provide the inference script for YUME 1.5—the only issue is that there are some descriptive errors in the README file. Specifically, the inference script for the 5B model should be scripts/inference/sample_5b.sh, rather than scripts/inference/sample_tts.sh. At present, I have successfully run the T2V function of this script; however, due to GPU memory limitations, I have only managed to implement two rounds of autoregressive generation. As an additional note, the I2V module in the inference code leverages “InternVL3-2B-Instruct” for image processing, which needs to be downloaded separately. If possible, I suggest the authors add this clarification to the README file and revise the description of the inference script for YUME 1.5.

Second, regarding the training scripts, they do seem to still be targeted at YUME 1.0. However, I noticed that the author provided a training file in Issue #16, though I haven’t had time to examine it in detail yet. Next, I will attempt to reproduce the training process of YUME 1.5 with the help of these resources. That said, it would be greatly appreciated if the author could release the complete training workflow.

Thank you very much for pointing out the error in the README. With community support, the Yume codebase will gradually improve.

[wine-plum]

Apologies. As I mentioned earlier, I do not have access to any GPUs at the moment, so I cannot currently provide code that is guaranteed to run bug-free.

I have provided training code in Issue #16 that works in theory. If you encounter any issues during training, please let me know, and I will try to fix them.

Additionally, regarding the autoregressive generation stopping after just 2 steps—how much VRAM does your GPU have? I am investigating the issue, and a possible cause is that the current sampling code is running in FP32 inference. However, I was previously able to reach about 5 steps on the webapp_single_gpu.py using a 16GB GPU.

Thank you for your reply!
Regarding the inference implementation: I used 8 NVIDIA A40 GPUs, with approximately 46GB of actual available VRAM per gpu.
The results of several inference tests are as follows:

1. I2V mode with "InternVL3-2B-Instruct" loaded: Failed to run entirely and directly threw an out-of-memory (OOM) error.
2. I2V mode without loading "InternVL3-2B-Instruct": Specified a random prompt to replace the output of the "InternVL3-2B-Instruct" model. This setup ran 7 rounds of autoregressive generation before throwing an OOM error.
   (Note: It seems that the "InternVL3-2B-Instruct" model is no longer needed after generating the prompt with it, so the model should be unloaded to free up VRAM—though I haven’t attempted to implement this yet.)
3. T2V mode with "InternVL3-2B-Instruct" loaded: Ran 2 rounds of autoregressive generation before throwing an OOM error.
4. T2V mode without loading "InternVL3-2B-Instruct": Ran 7 rounds of autoregressive generation before throwing an OOM error.
   (Note: The "InternVL3-2B-Instruct" model is completely unnecessary for T2V mode. It is only loaded in the code, and can simply be commented out for actual use.)

Compared with the original repository, I only modified the parts related to "InternVL3-2B-Instruct" in sample_5b.py for scenarios 2 and 4 mentioned above; all other content remained unchanged.

Attached here is the content of the inference script sample_5b.sh I used.

I2V mode:

#!/usr/bin/bash	

# DATA_DIR=./data
# IP=[MASTER NODE IP]
export TOKENIZERS_PARALLELISM=false

torchrun --nproc_per_node 8 --master_port 29717 \
    fastvideo/sample/sample_5b.py \
    --seed 43 \
    --gradient_checkpointing \
    --train_batch_size=1 \
    --max_sample_steps=600000 \
    --mixed_precision="bf16" \
    --allow_tf32 \
    --video_output_dir="./outputs" \
    --caption_path="./caption_re.txt" \
    --test_data_dir="./val" \
    --num_euler_timesteps 4 \
    --rand_num_img 0.6 \
    --t5_cpu \
    --jpg_dir="./jpg/"

T2V mode:

#!/usr/bin/bash	

# DATA_DIR=./data
# IP=[MASTER NODE IP]
export TOKENIZERS_PARALLELISM=false

torchrun --nproc_per_node 8 --master_port 29717 \
    fastvideo/sample/sample_5b.py \
    --seed 43 \
    --gradient_checkpointing \
    --train_batch_size=1 \
    --max_sample_steps=600000 \
    --mixed_precision="bf16" \
    --allow_tf32 \
    --video_output_dir="./outputs" \
    --caption_path="./caption_re.txt" \
    --test_data_dir="./val" \
    --num_euler_timesteps 4 \
    --rand_num_img 0.6 \
    --t5_cpu \
    --T2V \
    --prompt "A stylish woman walks down a Tokyo street filled with warm glowing neon and animated city signage."