Haibin He1, Maoyuan Ye1, Jing Zhang1, Juhua Liu1, Yong Luo1, Bo Du1
1School of Computer Science, Institute of Artificial Intelligence, Wuhan University
Overview | News | Installation | Training | Evaluation | Citation | Acknowledgements |
We conduct an upper-bound analysis through frame-wise question answering, counting a sample as correct if any frame yields the right answer, which significantly outperforms direct video-based inference and reveals a substantial performance gap. The results highlight the importance of identifying keyframes that contain question-relevant evidence for accurate Video TextVQA. Building on this insight, we propose a question-guided agent framework that explicitly anchors the relevant keyframes before answering. The approach operates effectively in a training-free setting and consistently surpasses direct video inference. With additional supervised fine-tuning (SFT) and reinforcement learning (RL), it achieves new state-of-the-art results.
Our contributions are three-fold:
-
We conduct an oracle upper-bound analysis that identifies evidence localization as the primary bottleneck in Video TextVQA.
-
We propose **VTAgent **, an agentic locate-and-focus framework via two steps: keyframe anchoring and keyframe-conditioned reasoning, enabling explicit evidence localization and grounded answer generation.
-
Comprehensive evaluations across benchmarks validate the effectiveness of VTAgent, yielding consistent gains in both training-free and fine-tuned settings and advancing the state-of-the-art in Video TextVQA.
-
2026-07-30: 📑 We released code and models of **VTAgent **. -
2026-05-06: 📑 We released our initial ArXiv paper.
We recommend creating separate virtual environments for the SFT (LLaMA-Factory), RL (verl), and Inference (Qwen3-VL + vllm) stages, and setting up each environment according to their corresponding instructions.
### SFT
# 1. Prepare sft data
# 2. Ensure that the file paths in vtagent_sft.json are correct, and properly configure the LLaMA-Factory/data/dataset_info.json and examples/qwen3vl_lora_sft_vtagent.yaml files.
llamafactory-cli train examples/train_lora/qwen3vl_lora_sft_vtagent.yaml
### RL
# 1. Prepare RL data
# 2. Ensure that the file paths in vtagent_rl.json are correct, and properly configure the verl/examples/vtagent/run_vtagent.sh.
sh verl/examples/vtagent/run_vtagent.sh### Properly configure the verl/examples/vtagent/run_vtagent.sh
sh infer.shWe also provide our trained model checkpoints here.
If you find our work helpful, please consider citing:
@article{he2026vtagent,
title={VTAgent: Agentic Keyframe Anchoring for Evidence-Aware Video TextVQA},
author={He, Haibin and Ye, Maoyuan and Zhang, Jing and Liu, Juhua and Du, Bo},
journal={arXiv preprint arXiv:2605.04870},
year={2026}
}
This project is based on LLaMA-Factory, verl, Qwen3-VL and vllm. Thanks for their wonderful works.