we see cuda initialization warnings sometimes, and the process doesnt' crash til we actually train something. we should set up a tiny thing that must execute on GPU to cause early failure, and that can run in a second or so, so we can bail instantly instead of waiting for training to start.
we see cuda initialization warnings sometimes, and the process doesnt' crash til we actually train something. we should set up a tiny thing that must execute on GPU to cause early failure, and that can run in a second or so, so we can bail instantly instead of waiting for training to start.