Feat: Handle ggml compute failures without crashing the program

[stduhpf]

It's annoying when the library crashes the whole program when something goes wrong.

For example, when the user asks for an image with insane resolution (on hip backend), we get something like this:

[INFO ] stable-diffusion.cpp:2921 - generating image: 1/1 - seed 42
[ERROR] stable-diffusion.cpp\ggml_extend.hpp:75   - ggml_backend_cuda_buffer_type_alloc_buffer: allocating 25929092.46 MiB on device 0: cudaMalloc failed: out of memory
[ERROR] stable-diffusion.cpp\ggml_extend.hpp:75   - ggml_gallocr_reserve_n: failed to allocate ROCm0 buffer of size 27188624053504
[ERROR] stable-diffusion.cpp\ggml_extend.hpp:1649 - unet: failed to allocate the compute buffer
Exception Code: 0xC0000005
0x00007FF7795514E6, H:\stable-diffusion.cpp\buildhip\bin\sd.exe(0x00007FF779280000) + 0x2D14E6 byte(s)
0x00007FF779378BDA, H:\stable-diffusion.cpp\buildhip\bin\sd.exe(0x00007FF779280000) + 0xF8BDA byte(s)
0x00007FF77940CD34, H:\stable-diffusion.cpp\buildhip\bin\sd.exe(0x00007FF779280000) + 0x18CD34 byte(s)
0x00007FF7793FA47C, H:\stable-diffusion.cpp\buildhip\bin\sd.exe(0x00007FF779280000) + 0x17A47C byte(s)
0x00007FF77946F94B, H:\stable-diffusion.cpp\buildhip\bin\sd.exe(0x00007FF779280000) + 0x1EF94B byte(s)
0x00007FF77933A247, H:\stable-diffusion.cpp\buildhip\bin\sd.exe(0x00007FF779280000) + 0xBA247 byte(s)
0x00007FF77931055A, H:\stable-diffusion.cpp\buildhip\bin\sd.exe(0x00007FF779280000) + 0x9055A byte(s)
0x00007FF779314E3E, H:\stable-diffusion.cpp\buildhip\bin\sd.exe(0x00007FF779280000) + 0x94E3E byte(s)
0x00007FF77929EC7E, H:\stable-diffusion.cpp\buildhip\bin\sd.exe(0x00007FF779280000) + 0x1EC7E byte(s)
0x00007FF779D69DBC, H:\stable-diffusion.cpp\buildhip\bin\sd.exe(0x00007FF779280000) + 0xAE9DBC byte(s)
0x00007FFA11AC7374, C:\Windows\System32\KERNEL32.DLL(0x00007FFA11AB0000) + 0x17374 byte(s), BaseThreadInitThunk() + 0x14 byte(s)
0x00007FFA11CDCC91, C:\Windows\SYSTEM32\ntdll.dll(0x00007FFA11C90000) + 0x4CC91 byte(s), RtlUserThreadStart() + 0x21 byte(s)

With these changes:

[INFO ] stable-diffusion.cpp:2947 - generating image: 1/1 - seed 42
[ERROR] stable-diffusion.cpp\ggml_extend.hpp:75   - ggml_backend_cuda_buffer_type_alloc_buffer: allocating 25929092.46 MiB on device 0: cudaMalloc failed: out of memory
[ERROR] stable-diffusion.cpp\ggml_extend.hpp:75   - ggml_gallocr_reserve_n: failed to allocate ROCm0 buffer of size 27188624053504
[ERROR] stable-diffusion.cpp\ggml_extend.hpp:1649 - unet: failed to allocate the compute buffer
[ERROR] stable-diffusion.cpp\ggml_extend.hpp:1919 - unet alloc compute buffer failed
[ERROR] stable-diffusion.cpp:1676 - diffusion model compute failed
[ERROR] stable-diffusion.cpp:1810 - Diffusion model sampling failed
[ERROR] stable-diffusion.cpp:2992 - sampling for image 1/1 failed after 3.10s
[INFO ] stable-diffusion.cpp:3000 - generating 0 latent images completed, taking 3.11s
[INFO ] stable-diffusion.cpp:3003 - decoding 0 latents
[INFO ] stable-diffusion.cpp:3017 - decode_first_stage completed, taking 0.00s
[INFO ] stable-diffusion.cpp:3310 - generate_image completed in 3.33s

When using the CLI, it doesn't matter too much, but when using the API for a server for example, avoiding these kinds of crashes is important.

[SkutteOleg]

Would this prevent ggml aborts? I'm using this project as a library in a C# application, and the thing that crashes the entire application is often ggml aborts, so I had to patch them out

[stduhpf]

Would this prevent ggml aborts? I'm using this project as a library in a C# application, and the thing that crashes the entire application is often ggml aborts, so I had to patch them out

Hopefully it avoids some of them by not trying to continue further once something goes wrong. I can't guarentee it prevents all cases of hitting a GGML_ABORT.

[SkutteOleg]

SkutteOleg@6ccc671

this is how i approached avoiding crashes and ggml aborts in my application, if it's of any use

[fszontagh]

Maybe the best approach to avoid the ggml's abort() call is to don't avoid the abort() call. In my SD project i just start an external process and capture the error output when it happens and warn about the user in the UI. Then the app just restart a new and fresh process. Better than let to kill the entire app.

[stduhpf]

@SkutteOleg you solution seems pretty good too.

[SkutteOleg]

Maybe the best approach to avoid the ggml's abort() call is to don't avoid the abort() call. In my SD project i just start an external process and capture the error output when it happens and warn about the user in the UI. Then the app just restart a new and fresh process. Better than let to kill the entire app.

this sounds like it requires quite a bit of platform-specific work

[fszontagh]

Yes, but worth it. Maybe the hardest part the protocol which used to communicate the 2 process. And at least, the main app is fast and responsive.

[idostyle]

Unreachable after return

[leejet]

Thank you for your contribution.