AMD gfx906 support for llama.cpp

[toat]

Hi,

I would like to use localai as backend llm with my AMD mi50 gfx906 but AMD dropped the support.
I found this projet which add deprecated GFX906 to some ML Software.

I tried to build a custom backend base on docker.io/mixa3607/llama.cpp-gfx906:-rocm-7.2.3 without success. Did not find a clear how to build a custom backend.

Thank you for your help !

[jangatzke]

LocalAI uses the gRPC interface to talk to llama.cpp. Shouldn't this work with a remote instance of llama-server, too? This way you could juts use mixa3607/llama.cpp-gfx906 and would not have to build a custom backend. I would love to test this, but I did not find a way to configure the gRPC-Server address in the localAI documentation.

Edit:
As it seems (source https://localai.io/advanced/index.html) you need to add this to your Docker container configuration:

LLAMACPP_GRPC_SERVERS="address1:port,address2:port"

Still need to find out if mixa3607/llama.cpp-gfx906 has gRPC support enabled by default or how to enable it.

Edit2:

As it seems the existing dockerfile does not include "-DGGML_RPC=ON" as llama.cpp build flag. I will patch the dockerfile and do a local test.

[toat]

External backend configuration exemple there
With external backend we dont have all feature from localai like model swaping.

My goal is to build a custom backend llama.cpp base on works from mixa3607 to have all powerfull feature of localai.

From docs exemples for custom backends, the dockerfile build from source LLAMA_REPO but mixa3607 provide docker image.

I am looking on the Dockerfile to import python folder to localAi custom backend Makefile.