[https://nvbugs/6050483][fix] Pin diffusers to 0.37.1 to fix UniPC scheduler device mismatch by chang-l · Pull Request #13017 · NVIDIA/TensorRT-LLM

chang-l · 2026-04-14T00:01:05Z

Summary

Pin diffusers to 0.37.1 in VBench integration tests to work around a device-mismatch regression in UniPCMultistepScheduler.multistep_uni_p_bh_update introduced by huggingface/diffusers#13356
The upstream change replaced torch.tensor(rks, device=device) with torch.stack(rks) where rks is a list mixing CPU tensors (from self.sigmas, intentionally kept on CPU) with a CUDA tensor, causing RuntimeError: Expected all tensors to be on the same device
Fixes all 5 visual gen tests that failed in L0_PostMerge RuntimeError: Encountered an error when fetching new request: Inplace update to inference tensor outside InferenceMode is not allowed.You can make a clone to get a normal tensor before doing inplace update. #2636 (test_vbench_dimension_score_wan22_a14b_nvfp4, test_vbench_dimension_score_wan22_a14b_fp8, test_vbench_dimension_score_wan, test_online_benchmark, test_offline_benchmark)

Root Cause

The tests installed diffusers from git+https://github.com/huggingface/diffusers.git (bleeding edge). Diffusers commit b114620 (Apr 3, 2026) changed the UniPC scheduler to:

# NEW (broken) - rks has CPU tensors from self.sigmas loop + CUDA tensor from ones()
rks.append(torch.ones((), device=device))
rks = torch.stack(rks)   # CRASH: mixed CPU/CUDA

Previously:

# OLD (correct) - coerces entire list to target device
rks.append(1.0)
rks = torch.tensor(rks, device=device)

Test plan

Verified diffusers==0.37.1 has AutoencoderKLWan, FlowMatchEulerDiscreteScheduler, UniPCMultistepScheduler (all needed classes)
Reproduced crash with diffusers 0.38.0.dev0 on B200 (UniPC step 1 crashes)
Verified fix with diffusers 0.37.1 on B200 (all scheduler steps pass)
CI: /bot run --stage-list "DGX_B200-4_GPUs-PyTorch-Post-Merge-1, DGX_B200-4_GPUs-PyTorch-Post-Merge-2"

🤖 Generated with Claude Code

Summary by CodeRabbit

Tests
- Updated test dependency configuration to use a stable released version instead of development source, improving test consistency and reproducibility.

…heduler device mismatch The VBench integration tests install diffusers from git HEAD, which pulled in huggingface/diffusers#13356. That PR changed `UniPCMultistepScheduler.multistep_uni_p_bh_update` to use `torch.stack(rks)` on a list mixing CPU tensors (from `self.sigmas`, intentionally kept on CPU) with a CUDA tensor (`torch.ones((), device=device)`), causing `RuntimeError: Expected all tensors to be on the same device`. Pin diffusers to 0.37.1 (latest stable release) which uses the original correct pattern: `torch.tensor(rks, device=device)`. Verified on B200: - diffusers 0.38.0.dev0 → crashes at UniPC step 1 - diffusers 0.37.1 → passes all steps Signed-off-by: Chang Liu <9713593+chang-l@users.noreply.github.com>

coderabbitai · 2026-04-14T00:03:48Z

📝 Walkthrough

Walkthrough

The _visual_gen_deps test fixture dependency installation is modified to pin the diffusers package to a specific released version (0.37.1) instead of installing directly from the Hugging Face GitHub repository development source.

Changes

Cohort / File(s)	Summary
Test Fixture Configuration `tests/integration/defs/examples/test_visual_gen.py`	Updated diffusers dependency installation from git source (`git+https://github.com/huggingface/diffusers.git`) to fixed release version (`diffusers==0.37.1`).

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and specifically describes the main change: pinning diffusers to version 0.37.1 to fix a UniPC scheduler device mismatch issue.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Description check	✅ Passed	The PR description is comprehensive and follows the template structure with clear sections for Summary, Root Cause, and Test plan.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

chang-l · 2026-04-14T00:14:22Z

/bot run --stage-list "DGX_B200-4_GPUs-PyTorch-Post-Merge-1, DGX_B200-4_GPUs-PyTorch-Post-Merge-2"

tensorrt-cicd · 2026-04-14T00:21:03Z

PR_Github #43116 [ run ] triggered by Bot. Commit: afe37cb Link to invocation

tensorrt-cicd · 2026-04-14T02:27:40Z

PR_Github #43116 [ run ] completed with state SUCCESS. Commit: afe37cb
/LLM/main/L0_MergeRequest_PR pipeline #33750 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

zhenhuaw-me · 2026-04-14T02:33:32Z

tests/integration/defs/examples/test_visual_gen.py

+    # Pin diffusers to 0.37.1 to avoid device-mismatch regression in
+    # UniPCMultistepScheduler.multistep_uni_p_bh_update introduced by
+    # huggingface/diffusers#13356 (torch.stack on mixed CPU/CUDA rks).
+    llm_venv.run_cmd(["-m", "pip", "install", "diffusers==0.37.1"])


Do we need to ping https://github.com/NVIDIA/TensorRT-LLM/blob/main/requirements.txt#L7 ?

zhenhuaw-me · 2026-04-14T06:32:36Z

tests/integration/defs/examples/test_visual_gen.py

 def _visual_gen_deps(llm_venv):
    """Install av + diffusers + ffmpeg once per session (shared by all video-gen fixtures)."""
    llm_venv.run_cmd(["-m", "pip", "install", "av"])
-    llm_venv.run_cmd(["-m", "pip", "install", "git+https://github.com/huggingface/diffusers.git"])


And we need to unwaive the waived corresponding tests.

… waivers Remove 5 test waivers added for nvbug 6050483 so CI will exercise the previously-failing visual gen tests with the pinned diffusers version. Removed waivers: - examples/test_visual_gen.py::test_vbench_dimension_score_wan22_a14b_nvfp4 - examples/test_visual_gen.py::test_vbench_dimension_score_wan22_a14b_fp8 - examples/test_visual_gen.py::test_vbench_dimension_score_wan - visual_gen/test_visual_gen_benchmark.py::test_online_benchmark[openai-videos] - visual_gen/test_visual_gen_benchmark.py::test_offline_benchmark Signed-off-by: Chang Liu <9713593+chang-l@users.noreply.github.com>

chang-l requested a review from a team as a code owner April 14, 2026 00:01

github-actions bot assigned chang-l Apr 14, 2026

zhenhuaw-me reviewed Apr 14, 2026

View reviewed changes

chang-l requested a review from a team as a code owner April 14, 2026 16:33

chang-l force-pushed the fix/nvbug-6050483-pin-diffusers-unipc-scheduler branch from f8abb37 to dc7ae45 Compare April 14, 2026 16:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[https://nvbugs/6050483][fix] Pin diffusers to 0.37.1 to fix UniPC scheduler device mismatch#13017

[https://nvbugs/6050483][fix] Pin diffusers to 0.37.1 to fix UniPC scheduler device mismatch#13017
chang-l wants to merge 2 commits intoNVIDIA:mainfrom
chang-l:fix/nvbug-6050483-pin-diffusers-unipc-scheduler

chang-l commented Apr 14, 2026 •

edited

Loading

Uh oh!

coderabbitai bot commented Apr 14, 2026 •

edited

Loading

Walkthrough

Changes

Estimated code review effort

Uh oh!

chang-l commented Apr 14, 2026

Uh oh!

tensorrt-cicd commented Apr 14, 2026

Uh oh!

tensorrt-cicd commented Apr 14, 2026

Uh oh!

zhenhuaw-me Apr 14, 2026

Uh oh!

zhenhuaw-me Apr 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

chang-l commented Apr 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Root Cause

Test plan

Summary by CodeRabbit

Uh oh!

coderabbitai bot commented Apr 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Uh oh!

chang-l commented Apr 14, 2026

Uh oh!

tensorrt-cicd commented Apr 14, 2026

Uh oh!

tensorrt-cicd commented Apr 14, 2026

Uh oh!

zhenhuaw-me Apr 14, 2026

Choose a reason for hiding this comment

Uh oh!

zhenhuaw-me Apr 14, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

chang-l commented Apr 14, 2026 •

edited

Loading

coderabbitai bot commented Apr 14, 2026 •

edited

Loading