Regarding AHC local evaluation details

[rthapa84]

Awesome work. I have been playing around with AHC tasks (039 and 058), currently, not training, simply evolution based approach with similar setup as TTT (prompts, init programs, evaluators, local test cases), however, what I am noticing is, the improved program may get really good score locally, and runs within time limit, however, when submitted to official AtCoder, it times out or scores poorly. I wonder if you ran into this issue, and how did you address this? I am running this on this machine:

CPU(s): 384
Model name: AMD EPYC 9654 96-Core Processor
Thread(s) per core: 2

I know you mentioned in earlier post that this task could be super sensitive to machine type. Wanted to know how sensitive it was for you, and what exact machine did you run this on?

[koceja]

From my experience, there was not a large mismatch between the timeouts locally and on the official leaderboard. Make sure you are using the same platform (local docker container and submitted container) that we had listed in the paper. Ultimately there is going to be slight differences due to running on different machines, but I believe it should be close.

You can potentially explore tuning/lowering the timeout values in this file though: https://github.com/test-time-training/discover/blob/main/examples/ahc/lib/tool_wrappers/case_runner.py. You can debug this by printing out the duration of your programs too, I had done this to see why I was timing out. We used AHC's timeout mechanism that isn't completely foolproof unfortunately.

Here is some of the info on the CPUs we used.
vCPU(s): 1024
Model name: AMD EPYC 9J14 96-Core Processor
Hyperthreading for 188 v-cores per processor