How to inference the model on custom image?

[Ir1d]

Hi,

Thanks for the great work. When I try to inference the model and loading the checkpoints (CO3D), with whatever script I always get such errors: ModuleNotFoundError: No module named 'unsup3d.

@Brummi Could you please guide me on how to inference custom images using the pre-trained CO3D models? Thanks!

[Brummi]

Hi, thank you for your interest in our work! Could you share exactly which script you are using and maybe append a screenshot of the stacktrace? This sounds like a classical import issue. (Even though the module should be named derender3d) Best, Felix

[Ir1d]

Hi @Brummi Thanks for the prompt reply. I use torch.load() for the checkpoint on co3d, and it throws such error

[TouqeerAhmad]

Hi, I am getting the same error. I am trying to run the script as suggested in the evaluation python scripts/eval_cosy.py -- it is triggering when model is being tried to be loaded.

[Brummi]

Hi,
thank you for your interest!
This sounds like the model is trying to import the wrong module (it should try to import derender3d).

Could you maybe try this trick:

import sys
sys.modules['unsup3d'] = derender3d 

[Ir1d]

Thanks @Brummi I'll give it a try. Which script should I use for evaluating on my custom image? I want to try inferencing an image and get the normal

[TouqeerAhmad]

Can you please be specific in which file we should place this. I think the issue is with the load, I tried toch.load, and load with strict=False flag. Could not find any module named unsup3d anywhere in the other code files as well.

[Brummi]

The derender3d module was previously named unsup3d. So I guess this is some relict of refactoring the code for the github release. I am trying to check it right now.

[TouqeerAhmad]

Interestingly, I am able to load the model based on CelebA and run the co3d data with it by re-directing the checkpoint path:

cp_path = 'results/models/celebahq_nr/checkpoint005.pth'
model.load_model_state(torch.load(cp_path, map_location=device))

I get numbers like these as output:
19.8Hz Normal_l1: 1.14872 Normal_mse: 0.27270 Normal_dot: 0.40907 Normal_deviation: 47.53636 Albedo_sie: 0.07547 Albedo_l1: 0.85258 Albedo_ssim: 0.75887 Spec_l1: 0.11423 Spec_mse: 0.06137 Spec_sie: 0.04574

[Brummi]

Put this code snippet in the file where you want to load the CO3D checkpoint. This should fix the issue of it not knowing unsup3d.
I think the celebahq model doesnt import anything. To be honest, I am not entirely sure why the CO3D checkpoint requires imports.

import derender3d
import sys
sys.modules["unsup3d"] = derender3d

[Brummi]

@Ir1d I don't have a script where you can just load up an image and then inference on it. However, the scripts/images_decomposition_co3d.py file almost does what you want. It shows you how to do inference with the CO3D model on the CO3D dataset and then save all the outputs.

(Concretely, in line 241, the normal map is saved to disk)

[TouqeerAhmad]

Unfortunately the Evaluation script is misleading, it assumes everything is pre-computed and calculated the metrics. Better would have been a simple example of inferring all those i.e., step before running the metrics.

[TouqeerAhmad]

FYI: lines 12 and 14 need to be switched to get the scripts/images_decomposition_co3d.py running.