fix video mm

[tastelikefeet]

PR type

• Bug Fix
• New Feature
• Document Updates
• More Models or Datasets Support

PR information

Write the detail information belongs to this PR.

Experiment results

Paste your experiment result here(if needed).

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a bug in video multimedia processing by ensuring that video data is preprocessed using the correct dedicated function. Additionally, it refines the handling of multimodal input features by dynamically incorporating VLM-specific fields, which enhances the robustness and extensibility of the data transformation process for vision-language models.

Highlights

• Video Multimedia Processing: Corrected the function used for preprocessing video messages from preprocess_images to preprocess_videos in src/twinkle/template/base.py, ensuring proper handling of video data.
• VLM Data Handling: Modified the to_transformers_dict function in src/twinkle/processor/base.py to dynamically include VLM-specific concatenation fields, improving flexibility and correctness for multimodal inputs.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• src/twinkle/processor/base.py
  • Updated the _keys list in to_transformers_dict to remove hardcoded pixel_values and image_grid_thw.
  • Added InputProcessor.VLM_CONCAT_FIELDS to the _keys list for dynamic inclusion of VLM-specific fields.
• src/twinkle/template/base.py
  • Changed the function call from self.preprocess_images(msg_videos) to self.preprocess_videos(msg_videos) when processing video messages.

Activity

• The pull request was created by tastelikefeet.
• The pull request is marked as a bug fix.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request addresses a bug in multimedia video processing. The change in src/twinkle/template/base.py is a good fix, correctly using preprocess_videos for video data. The update in src/twinkle/processor/base.py improves robustness by using the VLM_CONCAT_FIELDS set, making the code more maintainable. I have left one comment suggesting a minor performance optimization. To ensure the stability of this fix, consider adding a unit test case for video data processing to tests/processor/test_processor.py.

[gemini-code-assist]

For better performance, this _keys list should be defined once outside the for _input in inputs: loop, preferably as a set for O(1) average time complexity for membership testing. It is constant for all iterations, so there is no need to reconstruct it every time.