How to solve CudaSuccess (30 vs. 0) error?

Hi, I flashed a TX2 with JetPack 3.0 and while running a process (a rosnode), I get the following error:

F0508 12:38:24.922801 24593 syncedmem.hpp:22] Check failed: error == cudaSuccess (30 vs. 0)  unknown error
*** Check failure stack trace: ***
    @       0x7fa6ced718  google::LogMessage::Fail()
    @       0x7fa6cef614  google::LogMessage::SendToLog()
    @       0x7fa6ced290  google::LogMessage::Flush()
    @       0x7fa6cefeb4  google::LogMessageFatal::~LogMessageFatal()
    @       0x7fb00635fc  caffe::SyncedMemory::mutable_cpu_data()
    @       0x7fafeec6a4  caffe::Blob<>::Reshape()
    @       0x7fafeecce4  caffe::Blob<>::Reshape()
    @       0x7faffd2af4  caffe::InputLayer<>::LayerSetUp()
    @       0x7fb00c9d28  caffe::Net<>::Init()
    @       0x7fb00cbf24  caffe::Net<>::Net()
    @       0x7fb00cf6dc  FRCNN_API::Detector::Set_Model()
    @           0x4b7614  FRCNN_API::Detector::Detector()
    @           0x4b4318  main
    @       0x7faf3368a0  __libc_start_main
Aborted (core dumped)

I scoured the internet for a while and the closest relevant thread I found was this GitHub issue: https://github.com/NVIDIA/DIGITS/issues/1663. I tried out the

sudo ldconfig /usr/local/cuda/lib64

but it didn’t work. When I tried out another suggestion posted there:

sudo apt-get install nvidia-modprobe

, I get an error saying

Unable to locate package nvidia-modprobe

.

Can someone please help me as to how should I go about solving this problem? Thanks.

Hi,

The solution you mentioned is for x86 environment and cannot be used on Jetson.

Guess that you don’t use the correct GPU architecture to compile Caffe.
For example, please add TX2 architecture(sm=62) into Makefile.config:

https://github.com/BVLC/caffe/blob/master/Makefile.config.example#L37

CUDA_ARCH := -gencode arch=compute_20,code=sm_20 \
		-gencode arch=compute_20,code=sm_21 \
		-gencode arch=compute_30,code=sm_30 \
		-gencode arch=compute_35,code=sm_35 \
		-gencode arch=compute_50,code=sm_50 \
		-gencode arch=compute_52,code=sm_52 \
		-gencode arch=compute_60,code=sm_60 \
		-gencode arch=compute_61,code=sm_61 \
		<b>-gencode arch=compute_62,code=sm_62 \</b>
		-gencode arch=compute_61,code=compute_61

Thanks.

Thanks for the help @AastaLLL, I modified Makefile.config file to add the TX2 architecture, recompiled it but I still get the same error.

Hi,

Please remember that it’s required to install OS and libraries from the same JetPack Installer.
The compatible CUDA driver for CUDA libraries is included in the OS image and cannot be found elsewhere.

We are checking this internally and will let you know the results later.
Thanks.

Hi,

Sorry, we didn’t notice that you are using ROS.
Could you check CUDA functionality with our official sample first?

/usr/local/cuda-8.0/bin/cuda-install-samples-8.0.sh .
cd NVIDIA_CUDA-8.0_Samples/0_Simple/vectorAdd
make
./vectorAdd

By the way, could you share the result you use JetPack3.0.
If possible, it’s recommended to use our latest JetPack3.2:

Thanks.

Hi, I ran the sample you provided. I got the following output:

[Vector addition of 50000 elements]
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
Copy output data from the CUDA device to the host memory
Test PASSED
Done

Could you please be more specific about which result should I share when using JetPack 3.0? The error I was getting is in the question itself.

Also, all of the libraries including the OS has been installed from the same JetPack (v3.0).
Thanks.

Hi,

Could you try to build Caffe with root authority?

sudo make

Thanks.

Hi rahulbohare1993,

Have you clarified the cause and resolved the error?
Any result can be shared?

Thanks