Hello, we want to benchmark with MIC in our work, while we also encountered several issue in reproduction the paper results (75.9 mIOU in GTA->cityscape).
We have review all previous threads #8, #9 about the reproduced issue.
To be honestly, we don't want to stick on the specific package version of mmcv or torch. Because the specific version of package one day will be deprecated away, the method and framework design itself should be robust across various version (the small variance of performance (<1~2% mIOU) could be acceptable). If the performance variance out of 2% range, it may show the bad case happened in a very bad way (hoeffding inequality).
Info field :
The following log shows the reproduced results only achieved 70.24 mIOU (6% dropping is huge)...orz.
20241018_065618.log
config :
241018_0655_gtaHR2csHR_mic_hrda_s2_baseline_29a2d.txt
env :
mic_env.txt
Finally, we're in the progress of running the code with fixed seed=1.
Hope the performance difference can be mitigated by this setup.
Any suggestion will be appreciated!!
Hello, we want to benchmark with MIC in our work, while we also encountered several issue in reproduction the paper results (
75.9 mIOUin GTA->cityscape).We have review all previous threads #8, #9 about the reproduced issue.
To be honestly, we don't want to stick on the specific package version of mmcv or torch. Because the specific version of package one day will be deprecated away, the method and framework design itself should be robust across various version (the small variance of performance (<1~2% mIOU) could be acceptable). If the performance variance out of 2% range, it may show the bad case happened in a very bad way (hoeffding inequality).
Info field :
The following log shows the reproduced results only achieved
70.24 mIOU(6% dropping is huge)...orz.20241018_065618.log
config :
241018_0655_gtaHR2csHR_mic_hrda_s2_baseline_29a2d.txt
env :
mic_env.txt
Finally, we're in the progress of running the code with fixed
seed=1.Hope the performance difference can be mitigated by this setup.
Any suggestion will be appreciated!!