Benchmark uses unsupported temperature parameter with reasoning models

[messimm]

Description

The benchmark currently passes the temperature parameter to all models by default.

However, reasoning models such as gpt-5 do not support custom temperature values and only allow the default value. As a result, evaluation fails with errors like:

Error code: 400 - Unsupported value: 'temperature' does not support 0.0 with this model.

gpt-5 is used as model by default.

Expected behavior

Do not send temperature (and similar sampling parameters) to reasoning models.

[123sio]

In our setup, temperature can be passed this way on the API platform we use, and the benchmark simply follows that interface.
You can find the specific API details here: https://yunwu.apifox.cn/api-232421952.

[messimm]

In our setup, temperature can be passed this way on the API platform we use, and the benchmark simply follows that interface.
You can find the specific API details here: https://yunwu.apifox.cn/api-232421952.

Thanks for the clarification!

I think there are two important points worth highlighting (possibly also documenting in the issue or benchmark description):

We are using different APIs. Even for the same underlying model and task, different API platforms (and their parameter handling) can lead to noticeably different results. This difference in API interfaces and implementations can affect benchmark comparability, so it might be useful to explicitly note this on GitHub.

Not everyone has access to Chinese API platforms. For this reason, it would be very interesting to see the results of Kandinsky 5.0 (2B and 19B) in the main leaderboard using your API setup, so that the results are directly comparable with other models evaluated through the same interface. Kandinsky 5.0 is currently the strongest open-source model on LMArena for video generation, so such a comparison could be quite valuable for the community.

I’d be very happy to collaborate on this. I can either provide pre-generated videos or share a link to the model along with detailed instructions for running it locally, depending on what works best for you.

Looking forward to your thoughts!