Question about answer parse

[Violettttee]

I find such a response by 4o:

"[PLEASE OUTPUT YOUR REASONING]\n\nFor the first image:\n- The insect has a single pair of wings, which is a defining characteristic of the order Diptera (flies).\n- The body structure, including the presence of halteres (reduced hindwings), further supports this classification.\n\nAnswer: Diptera\n\nFor the second image:\n- The insect has two pairs of wings, with the hindwings smaller than the forewings.\n- The body structure, including the narrow waist and antennae, is characteristic of the order Hymenoptera (wasps, bees, ants).\n\nAnswer: Hymenoptera"

i seems that i give it a example image and a query image, so the model response for each image with an answer.I wonder how to deal with this situation?
In the code i find it will just extract the first response after "Answer"

[woodfrog]

Yes, this seems to be a corner case, and it is a bit tricky to parse the final answer. In the current parsing implementation, we have this function to only keep the last "Answer:" pattern and should handle this corner case.

[Violettttee]

Yes, this seems to be a corner case, and it is a bit tricky to parse the final answer. In the current parsing implementation, we have this function to only keep the last "Answer:" pattern and should handle this corner case.

get～but does the latest leadboard use this function?since the score i have of gpt seems to be lower.
and i have another question is, i see in the code it set sampling config, and for those who exceeds the max limit, it will excape the generation. So i wonder does gpt4o-0513 trigger this?

[woodfrog]

get～but does the latest leadboard use this function?since the score i have of gpt seems to be lower.

The GPT-4o-0513 results on the leaderboard didn't use that function. But that function only captures a very small number of corner cases, like the example you showed. I ran our GPT-4o-0513 results with the latest parsing and metrics in this repo, and the Core set score is 0.01 points higher than the one on the leaderboard (52.66 vs. 52.65).

What score did you get? I uploaded some of our results used by the paper and headboard here, and you might refer to those. When we organized the data and uploaded them to HuggingFace, small instruction pieces were concatenated into long prompts by adding newlines as separators, making the query slightly different from the version we used for our paper. However, the response results should be very similar and won't affect the overall ranking trends. Is your score much lower than what we reported?

and i have another question is, i see in the code it set sampling config, and for those who exceeds the max limit, it will excape the generation. So i wonder does gpt4o-0513 trigger this?

GPT-4o and other flagship models with long context windows won't trigger this. This "escaping" will only be triggered for multi-image tasks (not including video tasks) when the number of query images exceeds the preset max_image_num config (which is tuned for each model). For video tasks, we use the max_image_num to sub-sample the video automatically, and the escaping is never triggered.