Warp CUDA error 3

[peastman]

I'm trying to use the finetune.py script for training a model. I run it with this command:

python finetune.py --data_path=data.db --base_model=orb_v3_conservative_omol --stress_loss_weight=0

It just keeps printing the following errors over and over until I stop it:

Warp CUDA error 3: initialization error (in function get_current_context, /builds/omniverse/warp/warp/native/warp.cu:355)
Warp CUDA error 3: initialization error (in function wp_cuda_context_set_current, /builds/omniverse/warp/warp/native/warp.cu:2134)

I'm using the version of orb-models from #155, CUDA 12.8, and warp-lang 1.11.1. I'm running on an H100.

Inference works fine. I only encounter this error with the training script.

[vsimkus]

Can you check the nvalchemiops version? I think they've release a new version today that breaks BC.

On the same branch, I've now limited to an earlier nvalchemiops version, so if you could test that - it would be great!

[peastman]

I have 0.2.0, which was released in December. I could try updating to the new version, but it sounds like you expect that to create problems, not fix them?

[yuyemin]

I got the same error with nvalchemi-toolkit-ops==0.2.0, warp-lang==1.12.0, and CUDA==12.9. Not sure if this warning would cause any harm to training.

[vsimkus]

I can reproduce the issue. The warnings are coming from inside the Dataloader subprocesses due to nvalchemiops. I will check if the newer version resolves them.

This should fix the spam:

import multiprocessing
multiprocessing.set_start_method("spawn", force=True)