fix(sdk): enable thinking mode on the llama.cpp VLM path#1161
Draft
Paul Zhu (vinovo) wants to merge 1 commit into
Draft
fix(sdk): enable thinking mode on the llama.cpp VLM path#1161Paul Zhu (vinovo) wants to merge 1 commit into
Paul Zhu (vinovo) wants to merge 1 commit into
Conversation
Paul Zhu (vinovo)
force-pushed
the
fix/gemma4-thinking
branch
2 times, most recently
from
41fbc01 to
5937047
Compare
Gemma-4 ships an mmproj file so it loads as a VLM and runs through
vlm.cpp, which hardcoded enable_thinking=false and dropped special
tokens in token_to_piece. Pass enable_thinking through and render
special tokens so reasoning channel delimiters reach the caller.
Closes qcom-ai-hub/geniex#1208
Signed-off-by: Paul Zhu <paulzhuzx@gmail.com>
Paul Zhu (vinovo)
force-pushed
the
fix/gemma4-thinking
branch
from
5937047 to
90e53b3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
geniex infer <gemma-4 gguf> --thinkproduced no thinking (<think>) content, even though the model supports reasoning and llama-cli renders it with the same GGUF.Gemma-4 ships an mmproj file, so it loads as a VLM and runs through
sdk/plugins/llama_cpp/src/vlm.cpp— notllm.cpp. That path suppressed thinking in two spots:LlamaVlm::apply_chat_templatehardcodedenable_thinking = false(and logged a "not supported" warning), so the chat template never emitted the<|think|>system-turn token that primes Gemma-4 to reason.LlamaVlm::generatecalledllama_token_to_piece(..., /*special=*/false), so the model's<|channel>/<channel|>reasoning delimiters rendered as empty strings and the CLI's<think>FSM never fired.This PR passes
enable_thinkingthrough on the VLM path and renders special tokens during generation (matching llama.cpp's ownmtmd-cli, which renders special tokens by default).Test plan
geniex infer google/gemma-4-E2B-it-qat-q4_0-gguf --thinkrenders a<think>…</think>block--think=falsesuppresses thinking (direct answer)--thinkstill renders<think>