ignore_mismatched_sizes=True

[ZhixuanWuA]

You may consider adding ignore_mismatched_sizes=True in the model from_pretrained method.�

How to solve, thank you.

[llyx97]

Thank you for reaching out. Could you provide more details about your issue?

[ZhixuanWuA]

I donnot know the real reason for the issue. For the issue, I search each 'from_pretrained' in 'Video-LLaVA/run_video_llava/llava/model/builder.py�', add 'ignore_mismatched_sizes=True�' (i.e. Line124, I revise 'model = PeftModel.from_pretrained(model, model_path, ignore_mismatched_sizes=True)�'). Finally, it doesn't work, and also says 'You may consider adding ignore_mismatched_sizes=True in the model from_pretrained method.'

You are using a model of type LanguageBindimage to instantiate a model of type clip_vision model. This is not supported for all configurations of m odels and can yield errors.
Traceback (most recent call last):
File "Video-LLaVA/run_video_11ava/run_inference_tempcompass-py", line 84, in ‹module>
tokenizer, model, processor, context_len = load_pretrained_model (model path, None, model_name, load_8bit, load_4bit, device=device)
File "Video-LLaVA/run_video_llava/llava/model/builder.py", line 154, in load_pretrained_model image_tower. load_model()
File "Video-LLaVA/run_video_11ava/11ava/model/multimodal._encoder/clip_encoder.py", line 24, in load_model self.vision_tower = CLIPVisionModel. from_pretrained(self.vision_tower_name)
File "envs/videollava2/lib/python3.10/site-packages/transformers/modeling_utils-py", line 2903, in from_pretrained) = cls. load_pretrained model(
File "envs/videollava2/1ib/python3.10/site-packages/transformers/modeling_utils.py", line 3310, in _load_pretrained _mod el raise RuntimeError (f"Error(s) in loading state_dict for (model.class____name}:\n\t{error_msg)")
RuntimeError: Error(s) in loading state_dict for CLIPVisionModel:
size mismatch for vision_model.embeddings.class_embedding: copying a param with shape torch.Size([1024]) from checkpoint, the shape in curr ent model is torch.Size([7681).