What model was used in the paper ?

[Mohamed-Dhouib]

Hello, in the paper, you said that you are using LLaVA-Next-7B; however, there is no 7B version of LLaVA-Net. Can you refer me to the Hugging Face page of the model used and provide a .sh script to reproduce the paper's results?

[hasanar1f]

Thanks for reaching out! The pre-trained checkpoints can be found under this huggingface model collection.

Please refer to these scripts and lmms-eval documentation to reproduce our results.

[Mohamed-Dhouib]

So the llava next you are referring to in the paper is llava 1.6 ? If so which one please ?

[hasanar1f]

Yes. We have used llava-hf/llava-v1.6-vicuna-7b-hf and llava-hf/llava-v1.6-vicuna-13b-hf. Thanks for bringing it up. I will clarify this in the paper.

[Mohamed-Dhouib]

Ok thanks, I'm trying to implement your work on Llava Next to compare with mine, can you refer me to the function that takes the attention scores per crop and returns the unpruned set of tokens ?

[hasanar1f]

Please see the line 978-979 from here and the ViT_Attn_Hook function. Here, we get the attention of top and bottom ViT layer.

using that attention, we calculated the mask. And finally apply that mask here. The unpruned set of tokens is referred to as image_features before it goes into the pack_HiRED_image_features function.

[Mohamed-Dhouib]

OK thanks !