is leaderboard score hard or soft?

[cyberjunk]

Sorry if this has been answered somewhere already, I wasn't able to find anything. Are the published scores on your leader board calculated using the formula for hard restriction or soft restriction (both presented in your paper)?

[Genteki]

I also noticed it. I think it's soft restriction.

You may see Copilot in Excel model in both paper and Leaderboard. And the 20% ACC is judged by soft restriction.

[cyberjunk]

I think it's soft restriction.

Yes, I also think that it is soft but would love to see some confirmation from authors.

[Kaka23333]

Hi guys. Sorry for the ambiguity. Due to the high cost of evaluating on all three test cases on the leaderboard, we now use the first test case for each task and use soft restriction for evaluation on the leaderboard.

[liufofo77-hash]

I am confused, if we only use the first test case for each task, how can we use soft restriciton? Should we run many times for the first test case then use soft restriction?