Feature/add support for hunyuan 1 5

.github/skills/add-modular-pipeline/SKILL.md

@@ -94,6 +94,8 @@ The latent shape & space contract is documented in the `LatentPipelineDriver` do
 
 - **Public latents are unpacked**: 4-D `[B, C, H/vae, W/vae]` for image, 5-D `[B, C, T_lat, H/vae, W/vae]` for video. No model-specific sequence packing on the public surface.
 - **Public methods exchange `LatentArtifact`**, never raw tensors. The four forwardable methods (`encode_media`, `decode_latent`, `create_noise_latent`, `add_noise_to_latent`) return `LatentArtifact` (or `DecodeResult` for `decode_latent`) and accept input artifacts / media dataclasses. Build outputs via `_make_latent_artifact(...)`.
+- **`encode_media` for image and video drivers**: Prioritise modular implementation — verify whether a VAE encoder block exists for the pipeline before writing inline encode logic. Image-only drivers implement `encode_media` for `ImageMedia` and raise `NotImplementedError` for `VideoMedia`; video drivers raise for `ImageMedia`. Check the Variants checklist (V2V item) before deciding.
+- **`add_noise_to_latent` for image and video drivers**: I2I and V2V workflows (encoding a source video, adding noise at a given `strength`, then denoising) require a working `add_noise_to_latent`. Prioritise modular implementation and verify whether relevant blocks exist to achieve this flow. If no modular blocks exist, for flow-matching schedulers (`FlowMatchEulerDiscreteScheduler`) use `scheduler.scale_noise`; for DDPM-family schedulers use `scheduler.add_noise`. Only raise `NotImplementedError` when you have verified the underlying pipeline has no I2I for image pipelines or V2V support for video pipelines and have cited the diffusers source proving it.
 - **Forwardable signature is enforced**: those four methods MUST match [`FORWARDABLE_METHOD_POSITIONAL`](../../../modular_diffusion_nodes_library/latent_pipeline_drivers/_base_driver_forwardable_signature.py) exactly — same names, same order, no extra positional or kw-only parameters, no `*args`/`**kwargs`. `__init_subclass__` raises `TypeError` at import time otherwise. Route driver-specific tunables through the driver-namespaced sub-bag on `LatentArtifact.meta` (`META_DRIVER_KEY`, `read_driver_meta` in [`driver_types.py`](../../../modular_diffusion_nodes_library/latent_pipeline_drivers/driver_types.py); SDXL `_KIND_META_KEY` is the canonical precedent).
 - **Public latents are normalised** (~N(0, 1)). If the VAE config publishes per-channel `latents_mean` / `latents_std`, apply whitening `(z - mean) / std` inside `encode_media` and the inverse inside `decode_latent`. If it publishes only scalar `scaling_factor` (and optional `shift_factor`), the standard `(z - shift) * scaling` transform is sufficient — no whitening. Verify on the actual VAE config of the model you are adding; do not assume from family name. Mirror the closest existing driver.
 - **`source_shape` rides on the artifact / media dataclass** (pixel-space), not as a separate parameter. `LatentArtifact.source_shape`, `ImageMedia.source_shape`, `VideoMedia.source_shape`, `MaskMedia.source_shape` are the only sources of truth. Drivers translate to latent-space dims internally.
@@ -223,7 +225,7 @@ Produce a plan in this format:
 | encode_prompt | ModularPipeline.sub_blocks["text_encoder"] (or override) | exists at ...; declare extra required inputs (e.g. `prompt_2`, image conditioning) if the block needs them | enumerate every `required=True` `InputParam` |
 | decode_latent | ModularPipeline.sub_blocks["decode"] | exists at ... | `latents` ← `latent.to_torch(device, dtype)`; `output_type="pil"` |
 | create_noise_latent | ModularPipeline PrepareLatents step | ... | `height`/`width` ← `source_shape[-2:]`; `batch_size=1`; `num_images_per_prompt=1`; `generator` ← `generator_state.to_generator()` |
-| add_noise_to_latent | ModularPipeline Img2Img steps | ... | enumerate every `required=True` `InputParam` (see Modular-Blocks-First rule); e.g. `latents` ← `self.create_noise_latent(...).to_torch(device, dtype)`, `image_latents` ← `latent.to_torch(device, dtype)`, `batch_size` ← `image_latents.shape[0]`, `dtype` ← `_get_device_and_type()[1]`, `generator` ← `generator_state.to_generator()` |
+| add_noise_to_latent | ModularPipeline Img2Img / Vid2Vid noise injection steps | ... | enumerate every `required=True` `InputParam` (see Modular-Blocks-First rule); e.g. `latents` ← `self.create_noise_latent(...).to_torch(device, dtype)`, `image_latents` ← `latent.to_torch(device, dtype)`, `batch_size` ← `image_latents.shape[0]`, `dtype` ← `_get_device_and_type()[1]`, `generator` ← `generator_state.to_generator()`. For video pipelines: `image_latents` ← encoded video latent; noise API is `scheduler.scale_noise` (flow-matching) or `scheduler.add_noise` (DDPM) — verify against the scheduler class. |
 | denoise_latent | super() → DiffusionPipeline.__call__ | three blockers, see SKILL | n/a (delegated) |
 
 ## Provider classification (must match the Rule 1, Axis B approval)
@@ -264,6 +266,7 @@ State whether this pipeline class gets a NEW runtime-parameters class or reuses
 - ControlNet supported? [yes/no] — if yes, plan to apply Pattern A from `/add-pipeline-variants`
 - Inpaint supported? [yes/no] — if yes, plan to set `_inpaint_pipeline_class` (Pattern B)
 - Runtime pipe-swap needed (e.g. conditional variant)? [yes/no] — if yes, plan Pattern C
+- **Video-to-video (V2V) supported?** [yes/no] — **video drivers only**. Determines whether `encode_media(VideoMedia)` and `add_noise_to_latent` need real implementations instead of `NotImplementedError`. To answer yes: (a) the VAE must accept `[B, C, T, H, W]` multi-frame input, and (b) the scheduler must expose `scale_noise` or `add_noise` for video latents. Cite the diffusers source for both. If no, raise `NotImplementedError` with an inline comment citing the source that confirms V2V is unsupported.
 
 ## Runtime parameter defaults and tooltips (Rule 4 — every cell needs an upstream citation)
 | Param | Default value | Default source | Tooltip text source |

docs/nodes/pipeline_builder.md

@@ -38,7 +38,7 @@ Category: `ModularDiffusion/Pipeline`
 
 | Name | Type | Notes |
 | --- | --- | --- |
-| `provider` | choice | `Flux`, `Flux2`, `Stable Diffusion`, `Stable Diffusion 3`, `Qwen`, `Z-Image`, `LTX`, `LTX2`, `WAN`. Changing this swaps every parameter below. |
+| `provider` | choice | `Flux`, `Flux2`, `Stable Diffusion`, `Stable Diffusion 3`, `Qwen`, `Z-Image`, `HunyuanVideo 1.5`, `LTX`, `LTX2`, `WAN`. Changing this swaps every parameter below. |
 | `pipeline_type` | choice | Per-provider pipeline class (e.g. `FluxPipeline`, `WanImageToVideoPipeline`). Determines what the pipeline can do. |
 | `<model repo>` | HF repo picker | Hugging Face repo ID. Diffusers-format only — single-file `.safetensors` checkpoints are not loaded directly. |
 

modular_diffusion_nodes_library/latent_pipeline_drivers/base_driver.py

@@ -1,3 +1,4 @@
+import inspect
 from abc import ABC, abstractmethod
 from typing import Any, ClassVar, TypeVar
 
@@ -338,15 +339,20 @@ def denoise_latent(
             latents = self.prepare_input_latent(latents, source_shape)
             kwargs["latents"] = latents
 
-        kwargs.setdefault("height", source_shape[-2])
-        kwargs.setdefault("width", source_shape[-1])
-        generator = kwargs.pop("generator", generator_state.to_generator())
+        # check that the pipeline supports the kwargs we are passing in
+        pipe_call_params = inspect.signature(pipe.__call__).parameters
+        if "height" in pipe_call_params:
+            kwargs.setdefault("height", source_shape[-2])
+        if "width" in pipe_call_params:
+            kwargs.setdefault("width", source_shape[-1])
+        if "callback_on_step_end" in pipe_call_params:
+            kwargs["callback_on_step_end"] = callback
 
+        generator = kwargs.pop("generator", generator_state.to_generator())
         pipe_kwargs: dict[str, Any] = {
             **kwargs,
             "output_type": "latent",
             "num_inference_steps": num_inference_steps,
-            "callback_on_step_end": callback,
             "generator": generator,
         }
 

modular_diffusion_nodes_library/latent_pipeline_drivers/driver_factory.py

@@ -8,6 +8,12 @@
 from modular_diffusion_nodes_library.latent_pipeline_drivers.flux2_klein import Flux2KleinLatentPipelineDriver
 from modular_diffusion_nodes_library.latent_pipeline_drivers.flux_fill import FluxFillLatentPipelineDriver
 from modular_diffusion_nodes_library.latent_pipeline_drivers.flux_kontext import FluxKontextLatentPipelineDriver
+from modular_diffusion_nodes_library.latent_pipeline_drivers.hunyuan_video1_5 import (
+    HunyuanVideo15TextToVideoLatentPipelineDriver,
+)
+from modular_diffusion_nodes_library.latent_pipeline_drivers.hunyuan_video1_5_i2v import (
+    HunyuanVideo15ImageToVideoLatentPipelineDriver,
+)
 from modular_diffusion_nodes_library.latent_pipeline_drivers.ltx import LTXLatentPipelineDriver
 from modular_diffusion_nodes_library.latent_pipeline_drivers.ltx2 import LTX2PipelineDriver
 from modular_diffusion_nodes_library.latent_pipeline_drivers.qwen import QwenLatentPipelineDriver
@@ -27,6 +33,8 @@
 # Maps pipeline class name prefix to the corresponding driver class.
 _DRIVER_REGISTRY: dict[str, type[LatentPipelineDriver]] = {
     "FluxFillPipeline": FluxFillLatentPipelineDriver,
+    "HunyuanVideo15Pipeline": HunyuanVideo15TextToVideoLatentPipelineDriver,
+    "HunyuanVideo15ImageToVideoPipeline": HunyuanVideo15ImageToVideoLatentPipelineDriver,
     "FluxKontextPipeline": FluxKontextLatentPipelineDriver,
     "FluxPipeline": FluxLatentPipelineDriver,
     "Flux2Pipeline": Flux2LatentPipelineDriver,

modular_diffusion_nodes_library/latent_pipeline_drivers/hunyuan_video1_5.py

@@ -0,0 +1,257 @@
+import logging
+from typing import Any, ClassVar, cast, override
+
+import torch  # type: ignore[reportMissingImports]
+from diffusers.modular_pipelines.hunyuan_video1_5.before_denoise import (  # type: ignore[reportMissingImports]
+    HunyuanVideo15PrepareLatentsStep,
+    HunyuanVideo15SetTimestepsStep,
+)
+from diffusers.modular_pipelines.hunyuan_video1_5.decoders import (  # type: ignore[reportMissingImports]
+    HunyuanVideo15VaeDecoderStep,
+)
+from diffusers.modular_pipelines.hunyuan_video1_5.modular_blocks_hunyuan_video1_5 import (
+    HunyuanVideo15AutoBlocks,
+)
+from diffusers.modular_pipelines.hunyuan_video1_5.modular_pipeline import (  # type: ignore[reportMissingImports]
+    HunyuanVideo15ModularPipeline,
+)
+from diffusers.modular_pipelines.modular_pipeline import (  # type: ignore[reportMissingImports]
+    ModularPipeline,
+    ModularPipelineBlocks,
+    PipelineState,
+)
+from diffusers.modular_pipelines.modular_pipeline_utils import (  # type: ignore[reportMissingImports]
+    InputParam,
+    OutputParam,
+)
+from diffusers.pipelines.pipeline_utils import DiffusionPipeline  # type: ignore[reportMissingImports]
+
+from modular_diffusion_nodes_library.artifact_utils.inpaint_mask_artifact import InpaintMaskArtifact
+from modular_diffusion_nodes_library.artifact_utils.latent_artifact import LatentArtifact
+from modular_diffusion_nodes_library.latent_pipeline_drivers.base_driver import LatentPipelineDriver
+from modular_diffusion_nodes_library.latent_pipeline_drivers.driver_types import (
+    DecodeResult,
+    GeneratorState,
+    ImageMedia,
+    VideoMedia,
+)
+
+logger = logging.getLogger("modular_diffusers_nodes_library")
+
+
+class _HunyuanVideo15EncodeVideoStep(ModularPipelineBlocks):
+    """Encode a list of PIL frames into a normalised HunyuanVideo 1.5 video latent [B, C, T_lat, H/vsf, W/vsf]."""
+
+    model_name = "hunyuan_video1_5"
+
+    @property
+    def inputs(self) -> list[InputParam]:
+        return [InputParam("frames", required=True), InputParam("generator")]
+
+    @property
+    def intermediate_outputs(self) -> list[OutputParam]:
+        return [
+            OutputParam(
+                "video_latents",
+                type_hint=torch.Tensor,
+                description="Normalised video latent in the HunyuanVideo 1.5 VAE's output space.",
+            ),
+        ]
+
+    @torch.no_grad()
+    def __call__(
+        self, components: HunyuanVideo15ModularPipeline, state: PipelineState
+    ) -> tuple[HunyuanVideo15ModularPipeline, PipelineState]:
+        block_state = cast(Any, self.get_block_state(state))
+
+        device = components._execution_device
+        dtype = components.vae.dtype
+
+        frame_tensors = [components.video_processor.preprocess(frame) for frame in block_state.frames]
+        video_tensor = torch.stack([t.squeeze(0) for t in frame_tensors], dim=0)  # [T, C, H, W]
+        video_tensor = video_tensor.permute(1, 0, 2, 3).unsqueeze(0).to(device=device, dtype=dtype)  # [1, C, T, H, W]
+
+        latents = components.vae.encode(video_tensor).latent_dist.sample(generator=block_state.generator)
+        block_state.video_latents = latents * components.vae.config.scaling_factor
+
+        self.set_block_state(state, block_state)
+        return components, state
+
+
+class _HunyuanVideo15AddNoiseStep(ModularPipelineBlocks):
+    """Add flow-matching noise to a HunyuanVideo 1.5 video latent at a strength-derived timestep."""
+
+    model_name = "hunyuan_video1_5"
+
+    @property
+    def inputs(self) -> list[InputParam]:
+        return [
+            InputParam("latents", required=True),
+            InputParam("noise", required=True),
+            InputParam("num_inference_steps", required=True),
+            InputParam("strength", required=True),
+        ]
+
+    @property
+    def intermediate_outputs(self) -> list[OutputParam]:
+        return [
+            OutputParam(
+                "noisy_latents",
+                type_hint=torch.Tensor,
+                description="Latents with noise added at the strength-derived timestep.",
+            ),
+        ]
+
+    @torch.no_grad()
+    def __call__(
+        self, components: HunyuanVideo15ModularPipeline, state: PipelineState
+    ) -> tuple[HunyuanVideo15ModularPipeline, PipelineState]:
+        block_state = cast(Any, self.get_block_state(state))
+
+        _, state = HunyuanVideo15SetTimestepsStep()(components, state)  # type: ignore[reportOperatorIssue]
+        timesteps = state.values.get("timesteps")
+        if timesteps is None or len(timesteps) == 0:
+            raise ValueError("HunyuanVideo15SetTimestepsStep did not return valid timesteps.")
+
+        num_inference_steps = block_state.num_inference_steps
+        strength = block_state.strength
+        init_timestep = min(num_inference_steps * strength, num_inference_steps)
+        t_start = int(max(num_inference_steps - init_timestep, 0))
+        latent_timestep = timesteps[t_start * components.scheduler.order :][:1]
+
+        block_state.noisy_latents = components.scheduler.scale_noise(
+            block_state.latents, latent_timestep, block_state.noise
+        )
+
+        self.set_block_state(state, block_state)
+        return components, state
+
+
+class HunyuanVideo15TextToVideoLatentPipelineDriver(LatentPipelineDriver):
+    produces_video: ClassVar[bool] = True
+    # EXAMPLE_DOC_STRING in pipeline_hunyuan_video1_5.py specifies fps=15
+    video_fps: ClassVar[int] = 15
+
+    def __init__(self, pipe: DiffusionPipeline):
+        super().__init__(pipe)
+
+    @classmethod
+    @override
+    def can_make_control_pipe_from_standard(cls, control_net_model_lists: list[str] | str | None) -> bool:
+        return False
+
+    @override
+    def _create_modular_pipe(self) -> ModularPipeline:
+        return HunyuanVideo15AutoBlocks().init_pipeline()
+
+    @override
+    def _extract_latents_from_output(self, pipe_output: Any) -> torch.Tensor:
+        """HunyuanVideo pipelines return video frames under `.frames` instead of `.images`."""
+        return pipe_output.frames
+
+    @override
+    def create_noise_latent(self, source_shape: tuple[int, ...], generator_state: GeneratorState) -> LatentArtifact:
+        generator = generator_state.to_generator()
+        num_frames, height, width = source_shape[-3], source_shape[-2], source_shape[-1]
+        prepare_latents = HunyuanVideo15PrepareLatentsStep()
+        output_state = self._call_block(
+            prepare_latents,
+            height=height,
+            width=width,
+            num_frames=num_frames,
+            num_videos_per_prompt=1,
+            generator=generator,
+            batch_size=1,
+        )
+        latents = self._get_required(output_state, "latents", torch.Tensor)
+        return self._make_latent_artifact(
+            latents,
+            source_shape=source_shape,
+            meta=GeneratorState.from_generator(generator).as_meta(),
+        )
+
+    @override
+    def decode_latent(self, latent: LatentArtifact) -> DecodeResult:
+        """Decode a 5-D HunyuanVideo latent and return the video frames."""
+        device, dtype = self._get_device_and_type()
+        latents = latent.to_torch(device=device, dtype=dtype)
+        vae_decoder_step = HunyuanVideo15VaeDecoderStep()
+        output_state = self._call_block(vae_decoder_step, latents=latents, output_type="pil")
+        video_frames = self._get_required(output_state, "videos", list)[0]
+        return video_frames
+
+    @override
+    def encode_media(self, media: ImageMedia | VideoMedia, generator_state: GeneratorState) -> LatentArtifact:
+        if isinstance(media, ImageMedia):
+            raise NotImplementedError(
+                f"Pipeline '{self.pipe.__class__.__name__}' does not support image encoding. Use a video input instead."
+            )
+        generator = generator_state.to_generator()
+        output = self._call_block(_HunyuanVideo15EncodeVideoStep(), frames=media.frames, generator=generator)
+        return self._make_latent_artifact(
+            self._get_required(output, "video_latents", torch.Tensor),
+            source_shape=media.source_shape,
+        )
+
+    @override
+    def add_noise_to_latent(
+        self,
+        latent: LatentArtifact,
+        generator_state: GeneratorState,
+        num_inference_steps: int,
+        strength: float,
+    ) -> LatentArtifact:
+        device, dtype = self._get_device_and_type()
+        source_shape = latent.source_shape
+        latents = latent.to_torch(device=device, dtype=dtype)
+        noise_artifact = self.create_noise_latent(source_shape, generator_state)
+        noise = noise_artifact.to_torch(device=device, dtype=dtype)
+        noise_generator_state = GeneratorState.from_artifact(noise_artifact) or generator_state
+        output = self._call_block(
+            _HunyuanVideo15AddNoiseStep(),
+            latents=latents,
+            noise=noise,
+            num_inference_steps=num_inference_steps,
+            strength=strength,
+        )
+        result = self._get_required(output, "noisy_latents", torch.Tensor)
+        return self._make_latent_artifact(
+            result,
+            source_shape=source_shape,
+            upstream=latent,
+            meta=noise_generator_state.as_meta(),
+        )
+
+    @override
+    def denoise_latent(
+        self,
+        latent: LatentArtifact | InpaintMaskArtifact,
+        num_inference_steps: int,
+        generator_state: GeneratorState,
+        callback: Any = None,
+        start_step: int = 0,
+        end_step: int = -1,
+        return_fully_denoised: bool = False,
+        **kwargs: Any,
+    ) -> LatentArtifact:
+        update_kwargs = kwargs.copy()
+        update_kwargs.pop("media_gen_conditioning", None)
+
+        guidance_scale = update_kwargs.pop("guidance_scale", None)
+        if guidance_scale is not None:
+            self._pipe.guider.guidance_scale = guidance_scale
+
+        update_kwargs["num_frames"] = latent.source_shape[-3]
+
+        # HunyuanVideo15Pipeline.__call__ has no callback_on_step_end parameter,
+        # so execution status, live preview and mid-run cancellation are not available for this pipeline.
+        return super().denoise_latent(
+            latent,
+            num_inference_steps,
+            generator_state=generator_state,
+            callback=callback,
+            start_step=start_step,
+            end_step=end_step,
+            return_fully_denoised=return_fully_denoised,
+            **update_kwargs,
+        )