math_functions.cu:130] Check failed: status == CUBLAS_STATUS_SUCCESS (14 vs. 0) CUBLAS_STATUS_INTERNAL_ERROR

[takecareofbigboss]

math_functions.cu:130] Check failed: status == CUBLAS_STATUS_SUCCESS (14 vs. 0) CUBLAS_STATUS_INTERNAL_ERROR
*** Check failure stack trace: ***
@ 0x7f2a5c719b4d google::LogMessage::Fail()
@ 0x7f2a5c71db67 google::LogMessage::SendToLog()
@ 0x7f2a5c71b9e9 google::LogMessage::Flush()
@ 0x7f2a5c71bced google::LogMessageFatal::~LogMessageFatal()
@ 0x7f2a63d10256 caffe::caffe_gpu_dot<>()
@ 0x7f2a63cd4ffc caffe::SoftmaxWithLossLayer<>::Forward_gpu()
@ 0x7f2a63b108cc caffe::Net<>::ForwardFromTo()
@ 0x7f2a63b10ca7 caffe::Net<>::Forward()
@ 0x7f2a6398a308 caffe::Solver<>::Step()
@ 0x7f2a63ae78ea caffe::Worker<>::InternalThreadEntry()
@ 0x7f2a63afb0d0 caffe::InternalThread::entry()
@ 0x7f2a63afb8f6 boost::detail::thread_data<>::run()
@ 0x7f2a5800ee83 thread_proxy
@ 0x7f2a4abbb1c3 start_thread
@ 0x7f2a4a8ed12d __clone

when i use single gpu, the model can train normaly. But when I use multi-gpu to train "sh examples/FRCNN/zf/train_frcnn.sh", it reminds me that ...

[takecareofbigboss]

it occurs when i set this: gpu=all !!!!!!!!!!!!!!!!!!!!!!!!!!!!
so, your code can not train with multi gpu?

if [ ! -n "$1" ] ;then
echo "$1 is empty, default is 0"
gpu=all
else
echo "use $1-th gpu"
gpu=$1
fi