Question about the results generated from the code

[ningfeiwang]

Hello,

Thanks for sharing the source code for the great work, which can benefit a lot. I have a question about the results generated from the code.

First, I found some bugs from the jupyter code. For instance
# apply patch
tform_batch = get_random_tform_batch(len(image_batch))
Should it be
tform_batch = get_random_tform_batch(image_batch)
I think the input of this function should be images instead of a int number.

After fixing that, I found I cannot generate similar results reported in the paper. Mine is as follows (run for 5 times and select the best results). My mAP is very high (around 0.25). It looks very different compared to the zebra one in the paper and the notebook. I feel it may be because the randomness things for initialization or others. I am not very sure if there exists some other problems. Could you please help me on that? I think what I did is just change the path and the bugs I reported above, but can get a much worse results. Thanks a lot!

[erasaur]

I think you're on the right track. The first thing I would try is generating the unclipped patch to make sure the code works, it should work without many retries. Also make sure you are using the same YOLOv3 commit or there may have been changes which may require retuning of parameters. The process is quite stochastic and may just require more retries.

[ningfeiwang]

Thanks a lot for your quick reply and your advice. I make sure that I use the same YOLOv3 commit mentioned in the readme file. And then I tried the unclipped patch attack, which can achieve around 0.058 mAP after 5 epochs (try 3 times). I feel this results are still a little bit higher than the paper's. Not sure if there exists some problem in the source code. Do you have any suggestions instead of running multiple times? And do you have any estimations such as usually how many times it is needed?

[ningfeiwang]

Hello,

I tried multiple times but still cannot get the lower mAP. I also found some settings are different from the paper. For instance, the LR should be reduced after 300 iterations but not from the 1 iterations. Thus, I think the root cause could be that some settings are different. Could you please share the configuration in "run-2019-04-07_21:24" and some logs for me to further debug on that? If you can provide the folder, it would be great. Thanks a lot!

[erasaur]

sorry to hear that. unfortunately I don't have access to the machines used for this experiment anymore.