server: Bring back multimodal support

[ngxson]

Multimodal has been removed since #5882

Current llama.cpp multimodal roadmap

(update 9th april 2025)

• mtmd (MulTi-MoDal) library (top prio 🔥 )
  • Implement libmtmd: llava : introduce libmtmd #12849
  • Support more models via libmtmd (top prio 🔥 ) : mtmd : merge llava, gemma3 and minicpmv CLI into single llama-mtmd-cli #13012
  • Support M-RoPE models via libmtmd (Qwen2VL, Qwen2.5VL) : mtmd : add qwen2vl and qwen2.5vl #13141
  • Support audio input
  • Use smart pointer in clip.cpp to avoid mem leak: clip : use smart pointer (⚠️ breaking change) #12869
  • Add wrapper for stb_image to avoid polluting project with the big header file --> Probably don't need since we're already having some helper in libmtmd acting as wrapper for stb_image
  • Unify conversion scripts --> best case scenario: having convert_hf_to_gguf.py that can output both text + vision GGUF files --> introduced in convert : experimental support for --mmproj flag #13023
  • Remove BOI / EOI token embeddings from clip.cpp (used by glm-edge): clip : remove boi/eoi embeddings for GLM-edge model (⚠️ breaking change) #13081
  • Refactor documentations (find a way to reduce number of README files): llava : update documentations #13055
• Implement libmtmd in server API and server web UI (top prio 🔥 )
  • Publish first proposal: server : vision support via libmtmd #12898
  • User can upload image from UI ( + drag-and-drop)
  • Nice-to-have: Better KV caching strategy (TBD)
  • Nice-to-have: allow loading remote image (may come with security risk)
  • Update the security policy, make it clear that bugs related to 3rd party lib (like stb_image) should be reported to upstream, not in llama.cpp
• Unify all vision CLI (like minicpmv-cli, gemma3-cli, etc) into a single CLI
• Add deprecation notice for llava.h (we will remove libllava) and clip.h (clip is now internal-only)
• Experimental support for audio input: (wip) support ultravox audio input #12745
• (far in the future) implement llama_multimodal API that supports image, audio and more!

[ZhenyaPav]

Can someone advise me on what is the latest (working) commit before multimodal was removed?

[HAV0X1014]

It is rather annoying that multimodal support was removed for the server and has not been re-implemented for such a long time now (4 months?). Multimodal LLMs and interleaved image and text models are growing in capability recently, and not being able to run models that used to work before is unfortunate. Seemingly, the only way to restore this functionality is to downgrade to a version that loses support for most new models and improvements.

I am not trying to demand multimodal/llava support to return, but show that this feature on the server is missed.

[micoraweb]

Hello, is there still no multimodal support in llama-server ? According to ReadMe in LLaMA.cpp HTTP Server it should be supported?

How to use it with OpenAI API format?

[its-ven]

Have there been any updates on this?

[HAV0X1014]

So far, it appears that there hasn't been any updates. This really stinks because there were updates to llava recently to support new models.

[micoraweb]

So, this functionality seems to be unavailable for months and there is no hope to get it running? With all the amazing new models we could work with, such as MiniCPM, even Pixtral, etc. Can someone point us at a working server software that allows working with the newer multimodal models ? We just need something like llama-server that should run these multimodal GGUFs. Perhaps one of the llama.cpp files allows to run a server? It's also important to have a standard (OpenAI) API to support standard interactions.. It's so frustrating to wait months and months for such an important feature with no one even bothering to reply!