[strange] Dense captions are all filtered out

[countytown]

Hi, I have some problems when trying some examples by runing caption_anything/model.py.

The dense captions are all filtered by min_ppl_score and min_clip_score in parse_dense_caption(). I noticed that ppl_score is always -100.0 and clip_score is always 0.0, for example:
{'generated_captions': {'raw_caption': 'there is a girl holding a cat and a dog in her arms'}, 'crop_save_path': 'result/crop_1699081274.5122437.png', 'mask_save_path': 'result/mask_1699081274.5082428.png', 'mask': <PIL.Image.Image image mode=RGB size=512x320 at 0x1D552934B80>, 'bbox': array([ 0, 1, 511, 317]), 'area': 81409, 'context_captions': [], 'ppl_score': -100.0, 'clip_score': 0.0},...}

I think the caption is reasonable and should not have that low score, have you met the similar problem or some guess about this?

[countytown]

The problem has been solved. It is because the result.get('ppl_score', -100.) and result.get('clip_score', 0.) in model.py is failed and use the default value -100 and 0.0.

To solve that, you need to modify the inference() in caption_anything/captioner/blip.py if you use blip or in caption_anything/captioner/blip2.py if you use blip2, to make sure the returned value contains clip_score or ppl_score. For example:

    @torch.no_grad()
    def inference(self, image: Union[np.ndarray, Image.Image, str], filter=False, args={}):
        image = load_image(image, return_type="pil")
        inputs = self.processor(image, return_tensors="pt").to(self.device, self.torch_dtype)
        out = self.model.generate(**inputs, max_new_tokens=50)
        captions = self.processor.decode(out[0], skip_special_tokens=True).strip()
        
        result = {}
        if self.enable_filter and filter:
            clip_score = self.filter_caption(image, captions)
            print(clip_score,'ssssssssss')
            result['clip_score'] = clip_score
        result.update({'caption':captions})
        print(f"\nProcessed ImageCaptioning by BLIPCaptioner, Output Text: {captions}")
        # return {'caption': captions}
        return result