Difference JSON file lines and images

[ilirhajrullahu]

Hello MM-Eureka Team,

First of all, great work! I wanted to start reproducing the results.

But first I have a question: There are 54931 lines in the JSON file (is in line with what you have written in 2.2 in the paper).
But there are only 47363 training images (in train folders) and 68 eval images (eval folders).

Even if I add the 4844 lines of code in JSON where the blank_image.jpg occurs (for text-only inputs) there are still ca. 2-3k images missing.

Am I doing any mistake or are some images missing? Are there cases where you use the same image for different questions?

If there is something missing could you please provide the data? Or explain if I am misunderstanding?

Thank you and best regards :)

[LeoDu0314]

Hi, thanks for your question!

First, could you please confirm where you got your data from?

There should indeed be 500 evaluation images, as listed here:
https://github.com/ModalMinds/MM-EUREKA/tree/main/eval/k12/images

[ilirhajrullahu]

Hi,

I have used MMPR.zip, K12.zip and dataset.jsonl from here: https://huggingface.co/datasets/FanqingM/MM-Eureka-Dataset/tree/main

In these folders like for e.g. chartqa in MMPR you have validation dataset.

Yes I saw the 500 evaluation images in k12, but it still doesn't add up to 54931 including every data that is released. Please correct me if I'm wrong.

Thank you :)

[ilirhajrullahu]

Hello, is there any update on the missing files in the dataset? Thank you and best regards :) .