hardware: Intel iGPU, dGPU and NPU support

[xiangyang-95]

Overview

• Intel's Lunar Lake is releasing soon, which has CPU, NPU and iGPU in a single chip

Tasklist

• engine: Intel OpenVino #677
• feat: Add Intel GPU Support via SYCL on Linux platform cortex.llamacpp#107
• doc: add Intel MKL compilation for building library with server example cortex.llamacpp#99 from @xiangyang-95

Original Post

Problem
Unable to use Intel integrated GPU and discrete GPU to offload the model inferencing.

Success Criteria
Able to use Intel integrated GPU and discrete GPU to offload the model inferencing.

Additional context
I can add in the documentation to add Intel GPU support to nitro inference server.

[rahulunair]

I am trying to do the same, this would be incredibly useful with the latest Intel Core Ultra chips as well, as there is a unified cpu + GPU architecture.

With this, any user using jan or nitro directly on latest intel core ultra cpus will get a boost from the gpu

Options:

1. Intel acceleration support for llama.cpp using ipex-llm as a backend: https://ipex-llm.readthedocs.io/en/latest/doc/LLM/Quickstart/llama_cpp_quickstart.html

2. Direct compilation of llama.cpp with sycl bindings on device that support sycl: https://github.com/ggerganov/llama.cpp/blob/master/README-sycl.md

[xiangyang-95]

@rahulunair I have integrated the nitro server with Intel OneAPI SYCL Optimization on llama.cpp. Stay tuned for the pull request.

[freelerobot]

closing as dupe of #677

[dan-menlo]

I'm re-opening this, given our discussions with Intel. We should evaluate the following possibilities:

• llama.cpp through sycl (preferred)
• engine: Intel OpenVino #677 (which Intel recommended on their call)
• IPEX-LLM (less preferred to OpenVino)