LocAnyMed: Vision-Language Grounding for Multimodal Medical Images

Zihan Wang^1,†, Tong Liu^1,†, Zhiwei Wang¹, Tao Huang¹, Wentao Jiang¹,
Sihan Ma², Shanshan Ye³, Xiaohui Yang⁴, Jing Zhang^1,∗

¹ School of Computer Science, Wuhan University, Wuhan, China
² College of Computing and Data Science, Nanyang Technological University, Singapore
³ Australian Artificial Intelligence Institute, School of Computer Science, University of Technology Sydney, Sydney, Australia
⁴ Henan University, Kaifeng, China
^† Equal contribution    ^∗ Corresponding author

Update

• 2026-08: LocAnyMed project page prepared.
• The training and evaluation code, LocAnyMed-200K dataset, LocAnyMed-CoT-20K dataset, and model checkpoints will be released in subsequent updates.

Introduction

LocAnyMed adapts LocateAnything-3B to heterogeneous medical images through a unified visual-grounding formulation and modality-aware visual adaptation. The framework covers CT, ultrasound, X-ray, and optical medical images, and supports positive detection queries, point localization, and absent-target queries in one instruction format.

The maintained LocAnyMed method consists of three connected parts:

1. Unified medical supervision. LocAnyMed-200K contains 209,910 image-query-answer records spanning four modalities. The records preserve the original grounding target format, including boxes, points, and empty answers for negative queries.
2. Modality-aware visual MoE. The visual FFN uses a 2:2 shared-to-specific split with four static experts. The modality field (CT_ALL, Xray_ALL, Ultrasound_ALL, or Optical_ALL) selects one expert together with the shared branch; no learned router is used.
3. Rationale continuation with multiscale fusion. LocAnyMed-CoT-20K contains detection-only rationale records. The <think>...</think> rationale is followed by the original <ref> and <box> target, while the final three MoonViT blocks use parameter-free multiscale fusion.

Usage

This page currently presents the LocAnyMed method, benchmark results, and qualitative examples. The reproducible training and evaluation implementation, datasets, and checkpoints will be added in subsequent public updates.

Main Results

Box-grounding performance on Same-set Det. Val. and Cross-set Det. Test. Specialized detectors are trained on the LocAnyMed detection training split, while general-purpose vision-language models and grounding specialists are evaluated without medical fine-tuning. All values are percentages.

Category	Method	Same F1@0.50	Same F1@0.95	Same F1 Mean	Cross F1@0.50	Cross F1@0.95	Cross F1 Mean
Open-set detector	Grounding DINO	51.21	11.82	40.35	20.09	1.28	11.33
Closed-set detector	DETR-R50	45.73	10.98	35.75	12.91	0.43	6.88
Closed-set detector	EdgeCrafter	79.98	8.42	61.00	1.94	0.25	1.43
Closed-set detector	YOLO26	78.61	16.08	63.72	0.77	0.12	0.62
Closed-set detector	LW-DETR-DINOv2	81.34	17.00	64.78	1.32	0.20	0.99
Closed-set detector	LW-DETR-Objects365	82.58	18.71	66.25	1.41	0.21	1.08
Vision-language model	DeepSeek-VL2-Small	7.87	0.44	3.60	8.87	0.08	2.78
Vision-language model	MiMo-VL-7B	10.84	0.18	4.00	13.76	0.01	3.57
Grounding specialist	LocateAnything-3B	10.64	0.59	5.72	11.87	0.42	5.70
Vision-language model	Ovis2.5-2B	18.13	0.03	5.90	9.08	0.00	2.21
Grounding specialist	Rex-Omni-3B	15.48	0.69	9.77	10.39	0.11	4.97
Vision-language model	GPT-5.5	30.24	0.24	13.74	29.07	0.05	11.84
Vision-language model	Qwen3-VL-4B	29.82	9.70	18.36	25.89	5.24	12.17
Vision-language model	Qwen3-VL-32B	29.91	17.61	23.27	27.49	9.28	16.02
Vision-language model	Qwen3-VL-8B	36.03	18.60	25.91	26.42	8.48	14.86
Ours	LocAnyMed	85.59	37.43	69.15	33.63	1.58	19.12

Visualization

Ground-truth and predicted boxes are shown in green and blue. The examples cover CT, X-ray, ultrasound, and optical imaging.

Acknowledgements

LocAnyMed is built upon LocateAnything. We thank the LocateAnything team for releasing the training code and the pretrained LocateAnything-3B checkpoint.

• LocateAnything project page
• Official LocateAnything code
• LocateAnything-3B checkpoint

Citation

Please cite LocAnyMed and LocateAnything when using this repository.

@misc{wang2026locanymedvisionlanguagegroundingmultimodal,
      title={LocAnyMed: Vision-Language Grounding for Multimodal Medical Images},
      author={Zihan Wang and Tong Liu and Zhiwei Wang and Tao Huang and Wentao Jiang and Sihan Ma and Shanshan Ye and Xiaohui Yang and Jing Zhang},
      year={2026},
      eprint={2608.03322},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2608.03322},
}