[everyone] set up the MoCo v2 pretraining pipeline (RESISC-45)

[taeil]

• using RESISC-45 dataset

The goal is to familiarize us with the code.
Everyone should do at least one run.

• @SuryaTechie
• @oropezaev
• @TsungChinHanKen
• @taeil to verify tracking as well

[tchken]

• Confirmed the pre-trained run on RESISC via @Llano from my end
• haven't integrated the wandb

[suryagutta]

I am getting error:

(base) ➜  hpt git:(ernesto_branch) ✗cd OpenSelfSup
(base) ➜  OpenSelfSup git:(ernesto_branch) ✗.**/tools/dist_train.sh configs/selfsup/moco/r50_v2_resisc_in_basetrain_20ep.py 4**
/scratch/crguest/miniconda3/bin/python: Error while finding module specification for 'torch.distributed.launch' (ModuleNotFoundError: No module named 'torch')
(base) ➜  OpenSelfSup git:(master) ✗

[tchken]

Please make sure to run on hp120 by conda activate hp120. not on the base level

[suryagutta]

It works fine by following the steps:

1. Login (It will automatically take us to /scratch/crguest/hpt )
2. conda activate hp120
3. Then follow the steps from 'Verify Install' in the git

[tchken]

yep, Coloroda has setup the environment, we don't need to reinstal lnstall the torch and cuda ourselves (at lease on his machine)

[suryagutta]

I am getting the follwoing error (even after I tried to re-login):

File "tools/train.py", line 156, in <module>
    main()
  File "tools/train.py", line 93, in main
    init_dist(args.launcher, **cfg.dist_params)
  File "/scratch/crguest/miniconda3/envs/hp120/lib/python3.7/site-packages/mmcv/runner/dist_utils.py", line 17, in init_dist
    main()
  File "tools/train.py", line 93, in main
    _init_dist_pytorch(backend, **kwargs)
  File "/scratch/crguest/miniconda3/envs/hp120/lib/python3.7/site-packages/mmcv/runner/dist_utils.py", line 30, in _init_dist_pytorch
    init_dist(args.launcher, **cfg.dist_params)
  File "/scratch/crguest/miniconda3/envs/hp120/lib/python3.7/site-packages/mmcv/runner/dist_utils.py", line 17, in init_dist
    torch.cuda.set_device(rank % num_gpus)
ZeroDivisionError: integer division or modulo by zero
    _init_dist_pytorch(backend, **kwargs)
  File "/scratch/crguest/miniconda3/envs/hp120/lib/python3.7/site-packages/mmcv/runner/dist_utils.py", line 30, in _init_dist_pytorch
    torch.cuda.set_device(rank % num_gpus)
ZeroDivisionError: integer division or modulo by zero
Traceback (most recent call last):
  File "tools/train.py", line 156, in <module>

[oropezaev]

That's the same error. It's like it is not recognizing the number of GPUs

[suryagutta]

Something happened. It's not able to get the GPUs: No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'

[tchken]

torch.cuda.is_available() return False now, seems that has been modified..trying to see if passing CUDA_VISIBLE_DEVICES=0,1,2,3 would work

[suryagutta]

Tried CUDA_VISIBLE_DEVICES=0,1,2,3 ./tools/dist_train.sh configs/selfsup/moco/r50_v2_resisc_in_basetrain_20ep.py 4 . It still fails. I even exported CUDA_VISIBLE_DEVICES=0,1,2,3 . It didn't hep.

[suryagutta]

Found the following process which might be blocking:
0 S crguest 10782 10474 0 80 0 - 3555 pipe_w 15:41 pts/0 00:00:00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn --exclude-dir=.idea --exclude-dir=.tox python

Will try to kill and start again.

[suryagutta]

Did kill the above process as I am concerned that I might mess up the env.
Found the following online:
the wrong match due to pytorch version and cuda version. I try torch1.6.0 in cuda10 first and get the same error. I solve the problem by downgrading to torch 1.3.1

We are having torch 1.6.0 and cuda 10.2

[suryagutta]

torch.cuda.is_available() is false .
Online search: Sometimes, the issue can be resolved by restarting the system.