WARNING:torch.distributed.run:
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
*****************************************
Disabling PyTorch because PyTorch >= 2.1 is required but found 2.0.1
Disabling PyTorch because PyTorch >= 2.1 is required but found 2.0.1
Disabling PyTorch because PyTorch >= 2.1 is required but found 2.0.1
Disabling PyTorch because PyTorch >= 2.1 is required but found 2.0.1
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
/home/brcao/.conda/envs/star/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/brcao/.conda/envs/star/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/brcao/.conda/envs/star/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/brcao/.conda/envs/star/lib/python3.10/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
/home/brcao/.conda/envs/star/lib/python3.10/site-packages/torchvision/datapoints/__init__.py:12: UserWarning: The torchvision.datapoints and torchvision.transforms.v2 namespaces are still Beta. While we do not expect major breaking changes, some APIs may still change according to user feedback. Please submit any feedback you may have in this issue: https://github.com/pytorch/vision/issues/6753, and you can also check out https://github.com/pytorch/vision/issues/7319 to learn more about the APIs that we suspect might involve future changes. You can silence this warning by calling torchvision.disable_beta_transforms_warning().
warnings.warn(_BETA_TRANSFORMS_WARNING)
/home/brcao/.conda/envs/star/lib/python3.10/site-packages/torchvision/transforms/v2/__init__.py:54: UserWarning: The torchvision.datapoints and torchvision.transforms.v2 namespaces are still Beta. While we do not expect major breaking changes, some APIs may still change according to user feedback. Please submit any feedback you may have in this issue: https://github.com/pytorch/vision/issues/6753, and you can also check out https://github.com/pytorch/vision/issues/7319 to learn more about the APIs that we suspect might involve future changes. You can silence this warning by calling torchvision.disable_beta_transforms_warning().
warnings.warn(_BETA_TRANSFORMS_WARNING)
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
/home/brcao/.conda/envs/star/lib/python3.10/site-packages/torchvision/datapoints/__init__.py:12: UserWarning: The torchvision.datapoints and torchvision.transforms.v2 namespaces are still Beta. While we do not expect major breaking changes, some APIs may still change according to user feedback. Please submit any feedback you may have in this issue: https://github.com/pytorch/vision/issues/6753, and you can also check out https://github.com/pytorch/vision/issues/7319 to learn more about the APIs that we suspect might involve future changes. You can silence this warning by calling torchvision.disable_beta_transforms_warning().
warnings.warn(_BETA_TRANSFORMS_WARNING)
/home/brcao/.conda/envs/star/lib/python3.10/site-packages/torchvision/transforms/v2/__init__.py:54: UserWarning: The torchvision.datapoints and torchvision.transforms.v2 namespaces are still Beta. While we do not expect major breaking changes, some APIs may still change according to user feedback. Please submit any feedback you may have in this issue: https://github.com/pytorch/vision/issues/6753, and you can also check out https://github.com/pytorch/vision/issues/7319 to learn more about the APIs that we suspect might involve future changes. You can silence this warning by calling torchvision.disable_beta_transforms_warning().
warnings.warn(_BETA_TRANSFORMS_WARNING)
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
/home/brcao/.conda/envs/star/lib/python3.10/site-packages/torchvision/datapoints/__init__.py:12: UserWarning: The torchvision.datapoints and torchvision.transforms.v2 namespaces are still Beta. While we do not expect major breaking changes, some APIs may still change according to user feedback. Please submit any feedback you may have in this issue: https://github.com/pytorch/vision/issues/6753, and you can also check out https://github.com/pytorch/vision/issues/7319 to learn more about the APIs that we suspect might involve future changes. You can silence this warning by calling torchvision.disable_beta_transforms_warning().
warnings.warn(_BETA_TRANSFORMS_WARNING)
/home/brcao/.conda/envs/star/lib/python3.10/site-packages/torchvision/datapoints/__init__.py:12: UserWarning: The torchvision.datapoints and torchvision.transforms.v2 namespaces are still Beta. While we do not expect major breaking changes, some APIs may still change according to user feedback. Please submit any feedback you may have in this issue: https://github.com/pytorch/vision/issues/6753, and you can also check out https://github.com/pytorch/vision/issues/7319 to learn more about the APIs that we suspect might involve future changes. You can silence this warning by calling torchvision.disable_beta_transforms_warning().
warnings.warn(_BETA_TRANSFORMS_WARNING)
/home/brcao/.conda/envs/star/lib/python3.10/site-packages/torchvision/transforms/v2/__init__.py:54: UserWarning: The torchvision.datapoints and torchvision.transforms.v2 namespaces are still Beta. While we do not expect major breaking changes, some APIs may still change according to user feedback. Please submit any feedback you may have in this issue: https://github.com/pytorch/vision/issues/6753, and you can also check out https://github.com/pytorch/vision/issues/7319 to learn more about the APIs that we suspect might involve future changes. You can silence this warning by calling torchvision.disable_beta_transforms_warning().
warnings.warn(_BETA_TRANSFORMS_WARNING)
/home/brcao/.conda/envs/star/lib/python3.10/site-packages/torchvision/transforms/v2/__init__.py:54: UserWarning: The torchvision.datapoints and torchvision.transforms.v2 namespaces are still Beta. While we do not expect major breaking changes, some APIs may still change according to user feedback. Please submit any feedback you may have in this issue: https://github.com/pytorch/vision/issues/6753, and you can also check out https://github.com/pytorch/vision/issues/7319 to learn more about the APIs that we suspect might involve future changes. You can silence this warning by calling torchvision.disable_beta_transforms_warning().
warnings.warn(_BETA_TRANSFORMS_WARNING)
/home/brcao/.conda/envs/star/lib/python3.10/site-packages/accelerate/accelerator.py:530: UserWarning: `log_with=tensorboard` was passed but no supported trackers are currently installed.
warnings.warn(f"`log_with={log_with}` was passed but no supported trackers are currently installed.")
/home/brcao/.conda/envs/star/lib/python3.10/site-packages/accelerate/accelerator.py:530: UserWarning: `log_with=tensorboard` was passed but no supported trackers are currently installed.
warnings.warn(f"`log_with={log_with}` was passed but no supported trackers are currently installed.")
/home/brcao/.conda/envs/star/lib/python3.10/site-packages/accelerate/accelerator.py:530: UserWarning: `log_with=tensorboard` was passed but no supported trackers are currently installed.
warnings.warn(f"`log_with={log_with}` was passed but no supported trackers are currently installed.")
/home/brcao/.conda/envs/star/lib/python3.10/site-packages/accelerate/accelerator.py:530: UserWarning: `log_with=tensorboard` was passed but no supported trackers are currently installed.
warnings.warn(f"`log_with={log_with}` was passed but no supported trackers are currently installed.")
Detected kernel version 5.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
2025-09-14 16:21:06,092 - video_to_video - INFO - Build text encoder with CLIP
2025-09-14 16:21:06,116 - video_to_video - INFO - Build text encoder with CLIP
2025-09-14 16:21:06,460 - video_to_video - INFO - Build text encoder with CLIP
2025-09-14 16:21:06,783 - video_to_video - INFO - Build text encoder with CLIP
2025-09-14 16:21:20,718 - video_to_video - INFO - Load model path ./pretrained_weight/light_deg.pt, with local status <All keys matched successfully>
2025-09-14 16:21:20,725 - video_to_video - INFO - Total model params: 1.90 B
2025-09-14 16:21:20,726 - video_to_video - INFO - Build noise_scheduler with GaussianDiffusion
2025-09-14 16:21:20,995 - video_to_video - INFO - Build Temporal VAE
=================Optimize ControlNet ======================
input_blocks.0.1.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.0.1.transformer_blocks.0.local2.conv1.weight will be optimized
input_blocks.1.1.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.1.2.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.1.2.transformer_blocks.0.local2.conv1.weight will be optimized
input_blocks.2.1.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.2.2.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.2.2.transformer_blocks.0.local2.conv1.weight will be optimized
input_blocks.4.1.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.4.2.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.4.2.transformer_blocks.0.local2.conv1.weight will be optimized
input_blocks.5.1.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.5.2.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.5.2.transformer_blocks.0.local2.conv1.weight will be optimized
input_blocks.7.1.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.7.2.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.7.2.transformer_blocks.0.local2.conv1.weight will be optimized
input_blocks.8.1.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.8.2.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.8.2.transformer_blocks.0.local2.conv1.weight will be optimized
middle_block.1.transformer_blocks.0.local1.conv1.weight will be optimized
middle_block.2.transformer_blocks.0.local1.conv1.weight will be optimized
middle_block.2.transformer_blocks.0.local2.conv1.weight will be optimized
output_blocks.3.1.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.3.2.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.3.2.transformer_blocks.0.local2.conv1.weight will be optimized
output_blocks.4.1.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.4.2.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.4.2.transformer_blocks.0.local2.conv1.weight will be optimized
output_blocks.5.1.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.5.2.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.5.2.transformer_blocks.0.local2.conv1.weight will be optimized
output_blocks.6.1.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.6.2.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.6.2.transformer_blocks.0.local2.conv1.weight will be optimized
output_blocks.7.1.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.7.2.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.7.2.transformer_blocks.0.local2.conv1.weight will be optimized
output_blocks.8.1.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.8.2.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.8.2.transformer_blocks.0.local2.conv1.weight will be optimized
output_blocks.9.1.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.9.2.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.9.2.transformer_blocks.0.local2.conv1.weight will be optimized
output_blocks.10.1.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.10.2.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.10.2.transformer_blocks.0.local2.conv1.weight will be optimized
output_blocks.11.1.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.11.2.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.11.2.transformer_blocks.0.local2.conv1.weight will be optimized
VideoControlNet.input_blocks.0.1.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.0.1.transformer_blocks.0.local2.conv1.weight will be optimized
VideoControlNet.input_blocks.1.1.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.1.2.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.1.2.transformer_blocks.0.local2.conv1.weight will be optimized
VideoControlNet.input_blocks.2.1.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.2.2.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.2.2.transformer_blocks.0.local2.conv1.weight will be optimized
VideoControlNet.input_blocks.4.1.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.4.2.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.4.2.transformer_blocks.0.local2.conv1.weight will be optimized
VideoControlNet.input_blocks.5.1.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.5.2.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.5.2.transformer_blocks.0.local2.conv1.weight will be optimized
VideoControlNet.input_blocks.7.1.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.7.2.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.7.2.transformer_blocks.0.local2.conv1.weight will be optimized
VideoControlNet.input_blocks.8.1.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.8.2.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.8.2.transformer_blocks.0.local2.conv1.weight will be optimized
VideoControlNet.middle_block.1.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.middle_block.2.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.middle_block.2.transformer_blocks.0.local2.conv1.weight will be optimized
Total number of trainable parameters to optimize: 629.89 million
start to load optimizer...
2025-09-14 16:21:21,817 - video_to_video - INFO - Load model path ./pretrained_weight/light_deg.pt, with local status <All keys matched successfully>
2025-09-14 16:21:21,824 - video_to_video - INFO - Total model params: 1.90 B
2025-09-14 16:21:21,826 - video_to_video - INFO - Build noise_scheduler with GaussianDiffusion
2025-09-14 16:21:21,883 - video_to_video - INFO - Load model path ./pretrained_weight/light_deg.pt, with local status <All keys matched successfully>
2025-09-14 16:21:21,889 - video_to_video - INFO - Total model params: 1.90 B
2025-09-14 16:21:21,891 - video_to_video - INFO - Build noise_scheduler with GaussianDiffusion
2025-09-14 16:21:22,145 - video_to_video - INFO - Build Temporal VAE
=================Optimize ControlNet ======================
input_blocks.0.1.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.0.1.transformer_blocks.0.local2.conv1.weight will be optimized
input_blocks.1.1.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.1.2.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.1.2.transformer_blocks.0.local2.conv1.weight will be optimized
input_blocks.2.1.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.2.2.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.2.2.transformer_blocks.0.local2.conv1.weight will be optimized
input_blocks.4.1.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.4.2.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.4.2.transformer_blocks.0.local2.conv1.weight will be optimized
input_blocks.5.1.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.5.2.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.5.2.transformer_blocks.0.local2.conv1.weight will be optimized
input_blocks.7.1.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.7.2.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.7.2.transformer_blocks.0.local2.conv1.weight will be optimized
input_blocks.8.1.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.8.2.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.8.2.transformer_blocks.0.local2.conv1.weight will be optimized
middle_block.1.transformer_blocks.0.local1.conv1.weight will be optimized
middle_block.2.transformer_blocks.0.local1.conv1.weight will be optimized
middle_block.2.transformer_blocks.0.local2.conv1.weight will be optimized
output_blocks.3.1.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.3.2.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.3.2.transformer_blocks.0.local2.conv1.weight will be optimized
output_blocks.4.1.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.4.2.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.4.2.transformer_blocks.0.local2.conv1.weight will be optimized
output_blocks.5.1.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.5.2.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.5.2.transformer_blocks.0.local2.conv1.weight will be optimized
output_blocks.6.1.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.6.2.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.6.2.transformer_blocks.0.local2.conv1.weight will be optimized
output_blocks.7.1.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.7.2.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.7.2.transformer_blocks.0.local2.conv1.weight will be optimized
output_blocks.8.1.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.8.2.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.8.2.transformer_blocks.0.local2.conv1.weight will be optimized
output_blocks.9.1.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.9.2.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.9.2.transformer_blocks.0.local2.conv1.weight will be optimized
output_blocks.10.1.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.10.2.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.10.2.transformer_blocks.0.local2.conv1.weight will be optimized
output_blocks.11.1.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.11.2.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.11.2.transformer_blocks.0.local2.conv1.weight will be optimized
VideoControlNet.input_blocks.0.1.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.0.1.transformer_blocks.0.local2.conv1.weight will be optimized
VideoControlNet.input_blocks.1.1.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.1.2.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.1.2.transformer_blocks.0.local2.conv1.weight will be optimized
VideoControlNet.input_blocks.2.1.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.2.2.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.2.2.transformer_blocks.0.local2.conv1.weight will be optimized
VideoControlNet.input_blocks.4.1.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.4.2.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.4.2.transformer_blocks.0.local2.conv1.weight will be optimized
VideoControlNet.input_blocks.5.1.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.5.2.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.5.2.transformer_blocks.0.local2.conv1.weight will be optimized
VideoControlNet.input_blocks.7.1.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.7.2.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.7.2.transformer_blocks.0.local2.conv1.weight will be optimized
VideoControlNet.input_blocks.8.1.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.8.2.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.8.2.transformer_blocks.0.local2.conv1.weight will be optimized
VideoControlNet.middle_block.1.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.middle_block.2.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.middle_block.2.transformer_blocks.0.local2.conv1.weight will be optimized
Total number of trainable parameters to optimize: 629.89 million
start to load optimizer...
2025-09-14 16:21:22,214 - video_to_video - INFO - Build Temporal VAE
=================Optimize ControlNet ======================
input_blocks.0.1.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.0.1.transformer_blocks.0.local2.conv1.weight will be optimized
input_blocks.1.1.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.1.2.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.1.2.transformer_blocks.0.local2.conv1.weight will be optimized
input_blocks.2.1.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.2.2.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.2.2.transformer_blocks.0.local2.conv1.weight will be optimized
input_blocks.4.1.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.4.2.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.4.2.transformer_blocks.0.local2.conv1.weight will be optimized
input_blocks.5.1.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.5.2.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.5.2.transformer_blocks.0.local2.conv1.weight will be optimized
input_blocks.7.1.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.7.2.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.7.2.transformer_blocks.0.local2.conv1.weight will be optimized
input_blocks.8.1.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.8.2.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.8.2.transformer_blocks.0.local2.conv1.weight will be optimized
middle_block.1.transformer_blocks.0.local1.conv1.weight will be optimized
middle_block.2.transformer_blocks.0.local1.conv1.weight will be optimized
middle_block.2.transformer_blocks.0.local2.conv1.weight will be optimized
output_blocks.3.1.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.3.2.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.3.2.transformer_blocks.0.local2.conv1.weight will be optimized
output_blocks.4.1.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.4.2.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.4.2.transformer_blocks.0.local2.conv1.weight will be optimized
output_blocks.5.1.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.5.2.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.5.2.transformer_blocks.0.local2.conv1.weight will be optimized
output_blocks.6.1.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.6.2.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.6.2.transformer_blocks.0.local2.conv1.weight will be optimized
output_blocks.7.1.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.7.2.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.7.2.transformer_blocks.0.local2.conv1.weight will be optimized
output_blocks.8.1.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.8.2.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.8.2.transformer_blocks.0.local2.conv1.weight will be optimized
output_blocks.9.1.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.9.2.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.9.2.transformer_blocks.0.local2.conv1.weight will be optimized
output_blocks.10.1.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.10.2.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.10.2.transformer_blocks.0.local2.conv1.weight will be optimized
output_blocks.11.1.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.11.2.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.11.2.transformer_blocks.0.local2.conv1.weight will be optimized
VideoControlNet.input_blocks.0.1.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.0.1.transformer_blocks.0.local2.conv1.weight will be optimized
VideoControlNet.input_blocks.1.1.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.1.2.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.1.2.transformer_blocks.0.local2.conv1.weight will be optimized
VideoControlNet.input_blocks.2.1.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.2.2.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.2.2.transformer_blocks.0.local2.conv1.weight will be optimized
VideoControlNet.input_blocks.4.1.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.4.2.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.4.2.transformer_blocks.0.local2.conv1.weight will be optimized
VideoControlNet.input_blocks.5.1.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.5.2.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.5.2.transformer_blocks.0.local2.conv1.weight will be optimized
VideoControlNet.input_blocks.7.1.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.7.2.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.7.2.transformer_blocks.0.local2.conv1.weight will be optimized
VideoControlNet.input_blocks.8.1.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.8.2.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.8.2.transformer_blocks.0.local2.conv1.weight will be optimized
VideoControlNet.middle_block.1.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.middle_block.2.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.middle_block.2.transformer_blocks.0.local2.conv1.weight will be optimized
Total number of trainable parameters to optimize: 629.89 million
start to load optimizer...
2025-09-14 16:21:24,460 - video_to_video - INFO - Load model path ./pretrained_weight/light_deg.pt, with local status <All keys matched successfully>
2025-09-14 16:21:24,469 - video_to_video - INFO - Total model params: 1.90 B
2025-09-14 16:21:24,470 - video_to_video - INFO - Build noise_scheduler with GaussianDiffusion
2025-09-14 16:21:24,765 - video_to_video - INFO - Build Temporal VAE
=================Optimize ControlNet ======================
input_blocks.0.1.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.0.1.transformer_blocks.0.local2.conv1.weight will be optimized
input_blocks.1.1.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.1.2.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.1.2.transformer_blocks.0.local2.conv1.weight will be optimized
input_blocks.2.1.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.2.2.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.2.2.transformer_blocks.0.local2.conv1.weight will be optimized
input_blocks.4.1.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.4.2.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.4.2.transformer_blocks.0.local2.conv1.weight will be optimized
input_blocks.5.1.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.5.2.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.5.2.transformer_blocks.0.local2.conv1.weight will be optimized
input_blocks.7.1.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.7.2.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.7.2.transformer_blocks.0.local2.conv1.weight will be optimized
input_blocks.8.1.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.8.2.transformer_blocks.0.local1.conv1.weight will be optimized
input_blocks.8.2.transformer_blocks.0.local2.conv1.weight will be optimized
middle_block.1.transformer_blocks.0.local1.conv1.weight will be optimized
middle_block.2.transformer_blocks.0.local1.conv1.weight will be optimized
middle_block.2.transformer_blocks.0.local2.conv1.weight will be optimized
output_blocks.3.1.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.3.2.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.3.2.transformer_blocks.0.local2.conv1.weight will be optimized
output_blocks.4.1.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.4.2.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.4.2.transformer_blocks.0.local2.conv1.weight will be optimized
output_blocks.5.1.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.5.2.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.5.2.transformer_blocks.0.local2.conv1.weight will be optimized
output_blocks.6.1.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.6.2.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.6.2.transformer_blocks.0.local2.conv1.weight will be optimized
output_blocks.7.1.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.7.2.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.7.2.transformer_blocks.0.local2.conv1.weight will be optimized
output_blocks.8.1.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.8.2.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.8.2.transformer_blocks.0.local2.conv1.weight will be optimized
output_blocks.9.1.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.9.2.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.9.2.transformer_blocks.0.local2.conv1.weight will be optimized
output_blocks.10.1.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.10.2.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.10.2.transformer_blocks.0.local2.conv1.weight will be optimized
output_blocks.11.1.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.11.2.transformer_blocks.0.local1.conv1.weight will be optimized
output_blocks.11.2.transformer_blocks.0.local2.conv1.weight will be optimized
VideoControlNet.input_blocks.0.1.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.0.1.transformer_blocks.0.local2.conv1.weight will be optimized
VideoControlNet.input_blocks.1.1.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.1.2.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.1.2.transformer_blocks.0.local2.conv1.weight will be optimized
VideoControlNet.input_blocks.2.1.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.2.2.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.2.2.transformer_blocks.0.local2.conv1.weight will be optimized
VideoControlNet.input_blocks.4.1.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.4.2.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.4.2.transformer_blocks.0.local2.conv1.weight will be optimized
VideoControlNet.input_blocks.5.1.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.5.2.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.5.2.transformer_blocks.0.local2.conv1.weight will be optimized
VideoControlNet.input_blocks.7.1.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.7.2.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.7.2.transformer_blocks.0.local2.conv1.weight will be optimized
VideoControlNet.input_blocks.8.1.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.8.2.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.input_blocks.8.2.transformer_blocks.0.local2.conv1.weight will be optimized
VideoControlNet.middle_block.1.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.middle_block.2.transformer_blocks.0.local1.conv1.weight will be optimized
VideoControlNet.middle_block.2.transformer_blocks.0.local2.conv1.weight will be optimized
Total number of trainable parameters to optimize: 629.89 million
start to load optimizer...
[E ProcessGroupNCCL.cpp:828] [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLGATHER, Timeout(ms)=1800000) ran for 1808880 milliseconds before timing out.
Traceback (most recent call last):
File "/mnt/brcao/Repos/STAR/./video_super_resolution/scripts/train_sr.py", line 525, in <module>
model, optimizer, train_dataloader, lr_scheduler = accelerator.prepare(
File "/home/brcao/.conda/envs/star/lib/python3.10/site-packages/accelerate/accelerator.py", line 1555, in prepare
result = tuple(
File "/home/brcao/.conda/envs/star/lib/python3.10/site-packages/accelerate/accelerator.py", line 1556, in <genexpr>
self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)
File "/home/brcao/.conda/envs/star/lib/python3.10/site-packages/accelerate/accelerator.py", line 1398, in _prepare_one
return self.prepare_model(obj, device_placement=device_placement)
File "/home/brcao/.conda/envs/star/lib/python3.10/site-packages/accelerate/accelerator.py", line 1817, in prepare_model
model = torch.nn.parallel.DistributedDataParallel(
File "/home/brcao/.conda/envs/star/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 674, in __init__
_verify_param_shape_across_processes(self.process_group, parameters)
File "/home/brcao/.conda/envs/star/lib/python3.10/site-packages/torch/distributed/utils.py", line 118, in _verify_param_shape_across_processes
return dist._verify_params_across_processes(process_group, tensors, logger)
RuntimeError: DDP expects same model across all ranks, but Rank 0 has 2247 params, while rank 1 has inconsistent 0 params.
[E ProcessGroupNCCL.cpp:828] [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLGATHER, Timeout(ms)=1800000) ran for 1809018 milliseconds before timing out.
[E ProcessGroupNCCL.cpp:828] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLGATHER, Timeout(ms)=1800000) ran for 1809051 milliseconds before timing out.
[E ProcessGroupNCCL.cpp:828] [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLGATHER, Timeout(ms)=1800000) ran for 1809060 milliseconds before timing out.
Traceback (most recent call last):
File "/mnt/brcao/Repos/STAR/./video_super_resolution/scripts/train_sr.py", line 525, in <module>
model, optimizer, train_dataloader, lr_scheduler = accelerator.prepare(
File "/home/brcao/.conda/envs/star/lib/python3.10/site-packages/accelerate/accelerator.py", line 1555, in prepare
result = tuple(
File "/home/brcao/.conda/envs/star/lib/python3.10/site-packages/accelerate/accelerator.py", line 1556, in <genexpr>
self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)
File "/home/brcao/.conda/envs/star/lib/python3.10/site-packages/accelerate/accelerator.py", line 1398, in _prepare_one
return self.prepare_model(obj, device_placement=device_placement)
File "/home/brcao/.conda/envs/star/lib/python3.10/site-packages/accelerate/accelerator.py", line 1817, in prepare_model
model = torch.nn.parallel.DistributedDataParallel(
File "/home/brcao/.conda/envs/star/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 674, in __init__
_verify_param_shape_across_processes(self.process_group, parameters)
File "/home/brcao/.conda/envs/star/lib/python3.10/site-packages/torch/distributed/utils.py", line 118, in _verify_param_shape_across_processes
return dist._verify_params_across_processes(process_group, tensors, logger)
RuntimeError: DDP expects same model across all ranks, but Rank 1 has 2247 params, while rank 0 has inconsistent 0 params.
Traceback (most recent call last):
File "/mnt/brcao/Repos/STAR/./video_super_resolution/scripts/train_sr.py", line 525, in <module>
model, optimizer, train_dataloader, lr_scheduler = accelerator.prepare(
File "/home/brcao/.conda/envs/star/lib/python3.10/site-packages/accelerate/accelerator.py", line 1555, in prepare
result = tuple(
File "/home/brcao/.conda/envs/star/lib/python3.10/site-packages/accelerate/accelerator.py", line 1556, in <genexpr>
self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)
File "/home/brcao/.conda/envs/star/lib/python3.10/site-packages/accelerate/accelerator.py", line 1398, in _prepare_one
return self.prepare_model(obj, device_placement=device_placement)
File "/home/brcao/.conda/envs/star/lib/python3.10/site-packages/accelerate/accelerator.py", line 1817, in prepare_model
model = torch.nn.parallel.DistributedDataParallel(
File "/home/brcao/.conda/envs/star/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 674, in __init__
_verify_param_shape_across_processes(self.process_group, parameters)
File "/home/brcao/.conda/envs/star/lib/python3.10/site-packages/torch/distributed/utils.py", line 118, in _verify_param_shape_across_processes
return dist._verify_params_across_processes(process_group, tensors, logger)
RuntimeError: DDP expects same model across all ranks, but Rank 3 has 2247 params, while rank 0 has inconsistent 0 params.
Traceback (most recent call last):
File "/mnt/brcao/Repos/STAR/./video_super_resolution/scripts/train_sr.py", line 525, in <module>
model, optimizer, train_dataloader, lr_scheduler = accelerator.prepare(
File "/home/brcao/.conda/envs/star/lib/python3.10/site-packages/accelerate/accelerator.py", line 1555, in prepare
result = tuple(
File "/home/brcao/.conda/envs/star/lib/python3.10/site-packages/accelerate/accelerator.py", line 1556, in <genexpr>
self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)
File "/home/brcao/.conda/envs/star/lib/python3.10/site-packages/accelerate/accelerator.py", line 1398, in _prepare_one
return self.prepare_model(obj, device_placement=device_placement)
File "/home/brcao/.conda/envs/star/lib/python3.10/site-packages/accelerate/accelerator.py", line 1817, in prepare_model
model = torch.nn.parallel.DistributedDataParallel(
File "/home/brcao/.conda/envs/star/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 674, in __init__
_verify_param_shape_across_processes(self.process_group, parameters)
File "/home/brcao/.conda/envs/star/lib/python3.10/site-packages/torch/distributed/utils.py", line 118, in _verify_param_shape_across_processes
return dist._verify_params_across_processes(process_group, tensors, logger)
RuntimeError: DDP expects same model across all ranks, but Rank 2 has 2247 params, while rank 0 has inconsistent 0 params.
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 3966005) of binary: /home/brcao/.conda/envs/star/bin/python3.10
Traceback (most recent call last):
File "/home/brcao/.conda/envs/star/bin/torchrun", line 8, in <module>
sys.exit(main())
File "/home/brcao/.conda/envs/star/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
return f(*args, **kwargs)
File "/home/brcao/.conda/envs/star/lib/python3.10/site-packages/torch/distributed/run.py", line 794, in main
run(args)
File "/home/brcao/.conda/envs/star/lib/python3.10/site-packages/torch/distributed/run.py", line 785, in run
elastic_launch(
File "/home/brcao/.conda/envs/star/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/home/brcao/.conda/envs/star/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
./video_super_resolution/scripts/train_sr.py FAILED
------------------------------------------------------------
Failures:
[1]:
time : 2025-09-14_16:51:39
host : <HOST>
rank : 1 (local_rank: 1)
exitcode : 1 (pid: 3966006)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
time : 2025-09-14_16:51:39
host : <HOST>
rank : 2 (local_rank: 2)
exitcode : 1 (pid: 3966007)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[3]:
time : 2025-09-14_16:51:39
host : <HOST>
rank : 3 (local_rank: 3)
exitcode : 1 (pid: 3966008)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2025-09-14_16:51:39
host : <HOST>
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 3966005)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
