Eval bug: Qwen3-VL image embedding doesn't work

[lilydjwg]

Name and Version

version: 9786 (fdb2c11)
built with GNU 16.1.1 for Linux x86_64

Operating systems

Linux

GGML backends

Vulkan

Hardware

Ryzen 7 5700X + Radeon RX 6650 XT

Models

https://huggingface.co/DevQuasar/Qwen.Qwen3-VL-Embedding-2B-GGUF Q8_0

Problem description & steps to reproduce

Server: llama-server -m Qwen.Qwen3-VL-Embedding-2B.Q8_0.gguf -mm mmproj-Qwen.Qwen3-VL-Embedding-2B.f16.gguf --embedding

Client: curl http://127.0.0.1:8080/embedding -X POST --data @a.json

a.json generated with: echo '{"content": { "prompt_string": "Minecraft 截图", "multimodal_data": [ "'"$(base64 a.jpg|tr -d '\n')"'" ] }}' > a.json

a.jpg is a JPEG image converted from a screenshot.

Server response:

{"error":{"code":500,"message":"Failed to tokenize prompt","type":"server_error"}}

It works with text only prompts.

It also gives the same error with CPU backend.

First Bad Commit

This is a bug of a recent new feature: #15108

Relevant log output

Logs

0.06.427.270 E mtmd_tokenize: error: number of media markers in text (0) does not match number of bitmaps (1)
0.06.427.317 W srv    operator(): got exception: {"error":{"code":500,"message":"Failed to tokenize prompt","type":"server_error"}}

[TheOneWhoWill]

I tested this on my local pc and managed to replicate the issue. I also found that it was the tokenize function inside of process_mtmd_prompt() that passes the prompt directly without including the required media marker placeholder. It was expecting at least 1 in your case and found none. I wrote the code to fix this and I'm gonna make a pull request

[TheOneWhoWill]

Yeah idk when this pull request will be merged, first contribution and all. If you need a working version just compile from my fork