can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first. When attempting to call a diffusers sdxl pipeline after doing inference with the mmgp library active.

[Meatfucker]

I have implemented mmgp into a flux pipeline that can be called repeatedly without issue. However if I them attempt to call a sdxl pipeline, it throws that error. Ive attempted to move everything I can find to cpu to no avail. If I start with sdxl and mmgp itll generate once fine, but the second call of the pipeline returns the same error.

The sdxl error happens whether or not sdxl is called with mmgp, it only seems to matter that mmgp has been called once before.

[deepbeepmeep]

I dont think this related directly to mmgp : the error message you have reported occurs when you are trying to convert some pytorch tensors to a numpy array. the conversion can only be done if the tensor is already on the cpu. So you should have something like : tensor.cpu().numpy(). Otherwise there is no point moving every tensor to the cpu.

[Meatfucker]

Its just a normal stablediffusionxlpipeline from diffusers which works normally if mmgp is not used, and will work the first time if it is, but not the second. The only thing Ive done is use .to("cpu") wherever possible. Should I be leaving everything to cuda for mmgp?

If I invoke the pipeline without the mmpg library loaded itll infer and produce images. If I have the mmgp library imported and I use its offloading feature, that same pipeline will infer with mmgp once, but then crash with that error after the last inference step if I try the exact same call again. You should be able to reproduce it just by calling a sdxl pipeline with mmgp twice.

This is the SDXL function that if called twice is causing the issue, but works fine without the mmpg calls and I switch back the .to("cpu) back to cuda:

async def generate_sdxl(prompt,
                        width,
                        height,
                        steps,
                        batch_size,
                        image=None,
                        strength=None,
                        negative_prompt=None,
                        model_name=None,
                        lora_name=None,
                        controlnet_processor=None,
                        controlnet_image=None,
                        controlnet_conditioning=None,
                        ip_adapter_image=None,
                        ip_adapter_strength=None,
                        guidance_scale=None,
                        scheduler=None,
                        seed=None):
    kwargs = {}
    kwargs["prompt"] = prompt
    if negative_prompt is not None:
        kwargs["negative_prompt"] = negative_prompt
    kwargs["width"] = width if width is not None else 1024
    kwargs["height"] = height if height is not None else 1024
    kwargs["num_inference_steps"] = steps if steps is not None else 30
    kwargs["num_images_per_prompt"] = batch_size if batch_size is not None else 4
    kwargs["guidance_scale"] = guidance_scale if guidance_scale is not None else 5.0
    strength = strength if strength is not None else 0.7
    controlnet_conditioning = controlnet_conditioning if controlnet_conditioning is not None else 0.5
    ip_adapter_strength = ip_adapter_strength if ip_adapter_strength is not None else 0.6
    if model_name is None:
        model_name = "misri/zavychromaxl_v100"
    if seed is not None:
        generator = [torch.Generator(device="cpu").manual_seed(seed + i) for i in range(kwargs["num_images_per_prompt"])]
        kwargs["generator"] = generator

    generator, processed_image = await get_pipeline(model_name, image, controlnet_image, controlnet_processor)

    if scheduler is not None:
        match scheduler:
            case "DDIMScheduler":
                generator.scheduler = DDIMScheduler.from_config(generator.scheduler.config)
            case "DDPMScheduler":
                generator.scheduler = DDPMScheduler.from_config(generator.scheduler.config)
            case "PNDMScheduler":
                generator.scheduler = PNDMScheduler.from_config(generator.scheduler.config)
            case "LMSDiscreteScheduler":
                generator.scheduler = LMSDiscreteScheduler.from_config(generator.scheduler.config)
            case "EulerDiscreteScheduler":
                generator.scheduler = EulerDiscreteScheduler.from_config(generator.scheduler.config)
            case "HeunDiscreteScheduler":
                generator.scheduler = HeunDiscreteScheduler.from_config(generator.scheduler.config)
            case "EulerAncestralDiscreteScheduler":
                generator.scheduler = EulerAncestralDiscreteScheduler.from_config(generator.scheduler.config)
            case "DPMSolverMultistepScheduler":
                generator.scheduler = DPMSolverMultistepScheduler.from_config(generator.scheduler.config)
            case "DPMSolverSinglestepScheduler":
                generator.scheduler = DPMSolverSinglestepScheduler.from_config(generator.scheduler.config)
            case "KDPM2DiscreteScheduler":
                generator.scheduler = KDPM2DiscreteScheduler.from_config(generator.scheduler.config)
            case "KDPM2AncestralDiscreteScheduler":
                generator.scheduler = KDPM2AncestralDiscreteScheduler.from_config(generator.scheduler.config)
            case "DEISMultistepScheduler":
                generator.scheduler = DEISMultistepScheduler.from_config(generator.scheduler.config)
            case "UniPCMultistepScheduler":
                generator.scheduler = UniPCMultistepScheduler.from_config(generator.scheduler.config)
            case "DPMSolverSDEScheduler":
                generator.scheduler = DPMSolverSDEScheduler.from_config(generator.scheduler.config)

    if ip_adapter_image is not None:
        try:
            generator.load_ip_adapter("h94/IP-Adapter", subfolder="sdxl_models", weight_name="ip-adapter_sdxl.bin")
            generator.set_ip_adapter_scale(ip_adapter_strength)
            kwargs["ip_adapter_image"] = ip_adapter_image
        except Exception as e:
            print(f"SDXL IP ADAPTER ERROR: {e}")

    if lora_name is not None:
        lora_list = []
        for lora in lora_name:
            try:
                lora_name = os.path.splitext(lora)[0]
                generator.load_lora_weights(f"loras/sdxl/{lora}", adapter_name=lora_name)
                lora_list.append(lora_name)
            except Exception as e:
                print(f"SDXL LORA ERROR: {e}")
        generator.set_adapters(lora_list)

    offloaded_object = offload.all(generator, profile_type.LowRAM_LowVRAM)


    if image is not None:
        kwargs["image"] = image
        kwargs["strength"] = strength
        if controlnet_processor is not None:
            kwargs["control_image"] = processed_image
            kwargs["controlnet_conditioning_scale"] = controlnet_conditioning
        images = generator(**kwargs).images
    else:
        if controlnet_processor is not None:
            kwargs["image"] = processed_image
            kwargs["controlnet_conditioning_scale"] = controlnet_conditioning
        images = generator(**kwargs).images

    offloaded_object.unload_all()
    offloaded_object.release()

    del generator  #offloaded_object
    torch.cuda.empty_cache()
    gc.collect()

    return images

async def get_pipeline(model_name, image, controlnet_image, controlnet_processor):
    processed_image = None
    if image is not None:
        if controlnet_image is not None:
            controlnet, processed_image = await get_sdxl_controlnet(controlnet_processor, controlnet_image)
            generator = StableDiffusionXLControlNetImg2ImgPipeline.from_pretrained(model_name,
                                                                                   use_safetensors=True,
                                                                                   torch_dtype=torch.bfloat16,
                                                                                   controlnet=controlnet,
                                                                                   ).to("cpu")

        else:
            generator = StableDiffusionXLImg2ImgPipeline.from_pretrained(model_name,
                                                                         torch_dtype=torch.bfloat16,
                                                                         use_safetensors=True).to("cpu")

    else:
        if controlnet_image is not None:
            controlnet, processed_image = await get_sdxl_controlnet(controlnet_processor, controlnet_image)
            generator = StableDiffusionXLControlNetPipeline.from_pretrained(model_name,
                                                                            use_safetensors=True,
                                                                            torch_dtype=torch.bfloat16,
                                                                            controlnet=controlnet,
                                                                            ).to("cpu")

        else:
            generator = StableDiffusionXLPipeline.from_pretrained(model_name,
                                                                  torch_dtype=torch.bfloat16,
                                                                  use_safetensors=True).to("cpu")

    return generator, processed_image```

[deepbeepmeep]

beside the frompretrained(). you just need to add the cpu() on the line that causes the crash. as I dont have the crash stack I cant help

[Meatfucker]

My apologies, its being done through an api server so some of the crashes are not super helpful as its mostly uvicorn whining. Using prints, I have managed to narrow it down to the actual error happens during the second loading of the pipeline.

2025-06-21 11:14:59 | INFO     | sdxl_generate request received
Loading pipeline components...: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:02<00:00,  3.47it/s]
************ Memory Management for the GPU Poor (mmgp 3.4.9) by DeepBeepMeep ************
OFFLOADING SUCCESSFUL
 `last_sigmas_type='zero'` is not supported for `lower_order_final=False`. Changing scheduler {self.config} to have `lower_order_final` set to True.
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [00:29<00:00,  1.02it/s]
image generated
objects unloaded and released
returning images
2025-06-21 11:15:38 | INFO     | 127.0.0.1:56188 - "POST /sdxl_generate HTTP/1.1" 200
2025-06-21 11:15:43 | INFO     | sdxl_generate request received
Loading pipeline components...:  71%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏                                                | 5/7 [00:00<00:00,  8.79it/s]
2025-06-21 11:15:44 | INFO     | sdxl_generate ERROR: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.
2025-06-21 11:15:44 | INFO     | 127.0.0.1:38950 - "POST /sdxl_generate HTTP/1.1" 500
2025-06-21 11:15:44 | ERROR    | Exception in ASGI application

This is the crash and includes the successful call directly before it. I appreciate your time and am sorry to be annoying

[DenisKochetov]

@Meatfucker this is definitely a cache and memory issue - first run works fine, but on the second run the previous data stays cached and isn’t cleared, so you hit OOM. I’m facing the same problem right now and trying to fix it - any advice would be appreciated.