Update offload.py

[PeteHaughie]

Added logic check to see if cuda is actually available otherwise offload.py breaks the application via safetensors2.py. eg:

************ Memory Management for the GPU Poor (mmgp 3.1.4-15) by DeepBeepMeep ************
You have chosen a Medium speed profile that requires at least 32 GB of RAM and 24 GB of VRAM. Some VRAM is consuming just to make the model runs faster
Traceback (most recent call last):
  File "/Users/peterhaughie/Projects/YuEGP/inference/gradio_server.py", line 159, in <module>
    offload.profile(pipe, profile_no = profile,  compile = compile, quantizeTransformer= quantizeTransformer,  verboseLevel= args.verbose, **kwargs ) #pinnedMemory=False,
  File "/Users/peterhaughie/Projects/YuEGP/venv/lib/python3.10/site-packages/mmgp/offload.py", line 1758, in profile
    return all(pipe_or_dict_of_modules, verboseLevel = verboseLevel, **kwargs)
  File "/Users/peterhaughie/Projects/YuEGP/venv/lib/python3.10/site-packages/mmgp/offload.py", line 1413, in all
    self = offload()
  File "/Users/peterhaughie/Projects/YuEGP/venv/lib/python3.10/site-packages/mmgp/offload.py", line 996, in __init__
    self.device_mem_capacity = torch.cuda.get_device_properties(0).total_memory
  File "/Users/peterhaughie/Projects/YuEGP/venv/lib/python3.10/site-packages/torch/cuda/__init__.py", line 523, in get_device_properties
    _lazy_init()  # will define _get_device_properties
  File "/Users/peterhaughie/Projects/YuEGP/venv/lib/python3.10/site-packages/torch/cuda/__init__.py", line 310, in _lazy_init
    raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled

[PeteHaughie]

This whole file should be refactored to make it much more defensive - there are a lot of explicit calls for to("cuda") which need to either default to CPU or have a check for Cuda beforehand.

[deepbeepmeep]

thank you for you feedback. with all these changes have you managed to make mmgp work on a non Nvidia GPU ?

[PeteHaughie]

Not entirely no but I think there is a path towards it.it does however reliably start and not cause issues until you actually try to run a prompt.

…

On Wed, 2 Jul 2025, 2:57 am deepbeepmeep, ***@***.***> wrote: *deepbeepmeep* left a comment (deepbeepmeep/mmgp#13) <#13 (comment)> thank you for you feedback. with all these changes have you managed to make mmgp work on a non Nvidia GPU ? — Reply to this email directly, view it on GitHub <#13 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMR22ZWJ2VDJYR3FCEBGGL3GMU7HAVCNFSM6AAAAACAPIZVE2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTAMRVHE3DGNRQHE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[deepbeepmeep]

all right, so as soon as you have made it, I will merge your PR