Are new versions of koboldcpp broken with mmproj?

[Nogebator]

Versions after 1.93 or so stopped working with mmproj. Instead of a response, it returned chunks of hex code or incoherent fragments of sentences.

When playing Skyrim with the vision function, mmproj does not work at all.

Version 1.93 definitely works fine.

Used:
Mistral-Nemo-Instruct-2407-Q8_0.gguf

• pixtral-12b-mmproj-f16.gguf

[askmyteapot]

1.93 is over a year old.
Try one of the rolling builds to see if you still get the error.

https://github.com/LostRuins/koboldcpp/releases/tag/rolling

Terminal logs would help too, as well as what hardware you are using.

EDIT: That model is 2 years old now. There are much newer and more capable models available.

[Nogebator]

1.93 is over a year old. Try one of the rolling builds to see if you still get the error.

Yes, it's old. But it works. Newer versions don't work for some reason.

What I see in the log:

add_media: preproc_out has 1 entries, grid_x = 0, grid_y = 0, has_overview = 0
image_tokens->nx = 1055
image_tokens->ny = 1
batch_f32 size = 1
add_text: [IMG_END]

Processing Media Embedding 1 (1055 tokens)encoding image slice...
reserve_compute_meta:      CUDA0 compute buffer size =    83.63 MiB
reserve_compute_meta:        CPU compute buffer size =     3.01 MiB
reserve_compute_meta: graph splits = 1, nodes = 888
warmup: flash attention is disabled
clip_image_batch_encode: copying image 1/1 to input buffer (nx=512, ny=512)
clip_image_batch_encode: output embedding shape [5120, 1055, 1]
image slice encoded in 276 ms
decoding image batch 1/2, n_tokens_batch = 1024
image decoded (batch 1/2) in 74 ms
decoding image batch 2/2, n_tokens_batch = 31
image decoded (batch 2/2) in 57 ms
Processing Prompt [BATCH] (1161 / 1161 tokens)
Generating (2 / 4096 tokens)
(EOS token triggered! ID:2)
[18:04:20] CtxLimit:1163/8192, Init:0.09s, Processed:1161 in 0.51s (2299.01T/s), Generated:2/4096 in 0.03s (60.61T/s), Total:0.62s
Output: #

You can see that the model returned the answer "#." This is the most common response. Sometimes it can be a hex code or individual words. If you torture the model with repetitions for too long, it will very rarely produce meaningful text.

And specifically, Mistral + mmproj doesn't work. Cydonia works fine with or without mmproj.

[wbruna]

I can't reproduce it (Mistral-Nemo-Instruct-2407-Q4_K_S, mmproj-mistral-community_pixtral-12b-f16, Vulkan):

Could you post a full debug log?

[Nogebator]

Image description works for me too. It starts glitching when I speak to the model, like "Hello." It works fine without mmproj, but with mmproj, things start acting up.

[Nogebator]

Example of result:

[LostRuins]

@Nogebator can you provide links to the exact files you used for both the mmproj and model file

[Nogebator]

@LostRuins That's a tough question. I downloaded them a few years ago, and everything worked fine with older versions of koboldcpp. The problems arose when i updated koboldcpp.

But now i downloaded two Mistral models at random and tested them. The result was similar: an obvious crash.
https://huggingface.co/bartowski/Mistral-Nemo-Instruct-2407-GGUF/blob/main/Mistral-Nemo-Instruct-2407-Q8_0.gguf
https://huggingface.co/QuantFactory/Mistral-Nemo-Instruct-2407-abliterated-GGUF/blob/658180a6b6db6c2194da5a638b8cde50560dfd02/Mistral-Nemo-Instruct-2407-abliterated.Q8_0.gguf

And I tried these mmprojs:
https://huggingface.co/koboldcpp/mmproj/blob/main/pixtral-12b-mmproj-f16.gguf
https://huggingface.co/ggml-org/pixtral-12b-GGUF/blob/main/mmproj-pixtral-12b-f16.gguf

Sometimes, if i press retry for a long time, it might return a normal response. But more often, i get something like this:

PS: It looks like the crash is specifically with Mistral. The Cydonia model works fine.

[LostRuins]

I tried pixtral and it works fine for me. The model I used is from https://huggingface.co/bartowski/mistral-community_pixtral-12b-GGUF/resolve/main/mistral-community_pixtral-12b-Q2_K_L.gguf and https://huggingface.co/bartowski/mistral-community_pixtral-12b-GGUF/resolve/main/mmproj-mistral-community_pixtral-12b-f16.gguf

Could you give those models a try? They work for me.

[Nogebator]

@LostRuins Yes, this model works fine. Other models, like Cydonia (with its own mmproj), also work fine. It seems the crash only occurs with Mistral-Nemo and Mistral-Small models.

That's very sad. Mistral-Nemo, tuned by this author, was perfectly fine for me in terms of weight, speed, and content. I guess I'll have to look for another similar model. Or use koboldcpp v0.93 to work with this model.

Anyway, thanks for your reply.

[wbruna]

@Nogebator , does it still happen if you offload (say) a third of the model's layers, or load a smaller quant?

[LostRuins]

Searching the web, it seems like Mistral Nemo does not even support vision, and none of the models actually come with a mmproj. So I'd imagine it doesn't work and slapping a pixtral projector would just give unexpected results.

[Nogebator]

@LostRuins That's weird. I used it for over a year, and everything was fine. It still works fine with koboldcpp v0.93.

@wbruna I checked the Q4 version, and it doesn't work either. Reducing the number of loaded layers to 12 also didn't help.

UPD: By the way, I was wrong in my optimistic assessment of other models after a short test. A longer test showed that Cydonia now also works poorly with mmproj. Not always, but quite often, after one or two messages from startup, it crashes and starts returning empty responses.
This is even more unfortunate because Cydonia is a unique model, and there's nothing to replace it with. I used it for quite a while, and it worked perfectly well until the koboldcpp update.

UPD2: It looks like Cydonia's crash now depends on the "use ContextShift" function.