fix(python): fix VLM multi-turn image guard and test message order#1130
Merged
Conversation
Mengsheng Wu (mengshengwu)
requested review from
David Qian (Davidqian123) and
Paul Zhu (vinovo)
as code owners
4 tasks
Mengsheng Wu (mengshengwu)
force-pushed
the
worktree-fix-vlm-ci-tests
branch
from
18fe0c2 to
92e258d
Compare
_messages_have_modality scanned the entire conversation history, so the second turn of a multi-turn session incorrectly raised ValueError when generate(images=[]) was called with no new image (image already in the KV cache). Only the last user message carries new media; check that instead. Also align _vlm_prompt message content order with the C SDK reference (build_vlm_prompt in benchmark.c): text-first, then image, matching how vlm_message_to_common_chat_msg assembles the mtmd marker+text content. Signed-off-by: Mengsheng Wu <mengshengwu@outlook.com>
Cover the three predicate cases exercised by the fix: - prior-turn image/audio blocks must not gate the current text-only turn - current-turn media still trips the guard - empty message list is a no-op Signed-off-by: Mengsheng Wu <mengshengwu@outlook.com>
Mengsheng Wu (mengshengwu)
force-pushed
the
worktree-fix-vlm-ci-tests
branch
from
92e258d to
39b6c61
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1131
What
Two
test-sdk-qdc / llama_cpp_vlmfailures on X Elite Windows (CI run 28492301153):test_multi_turn_without_reset— raisedValueError: messages reference image content but generate(images=[...]) is empty._messages_have_modalityscanned the entire conversation history; the second turn passed the full history (containing the first-turn image block) toapply_chat_template, setting_last_template_has_image=True. The guard then blockedgenerate(images=[]), which is the correct call for a follow-up turn whose image is already in the KV cache.test_quality_keywords[npu]— image content order in_vlm_promptwas[image, text]. The C SDK reference (build_vlm_promptin benchmark.c, PR #1112) uses[text, image]order.Changes
modeling.py:_messages_have_modalitynow checks only the last user message instead of the full history. Prior-turn media is already in the KV cache and must not be re-supplied.test_llama_cpp_vlm.py: align_vlm_promptcontent order to[text, image], matchingbuild_vlm_promptin benchmark.c.test_error_messages.py: three unit regressions covering the modality-scope fix (ignore prior turns, detect current turn, empty list).