Hi, I flashed a TX2 with JetPack 3.0 and while running a process (a rosnode), I get the following error:
F0508 12:38:24.922801 24593 syncedmem.hpp:22] Check failed: error == cudaSuccess (30 vs. 0) unknown error
*** Check failure stack trace: ***
@ 0x7fa6ced718 google::LogMessage::Fail()
@ 0x7fa6cef614 google::LogMessage::SendToLog()
@ 0x7fa6ced290 google::LogMessage::Flush()
@ 0x7fa6cefeb4 google::LogMessageFatal::~LogMessageFatal()
@ 0x7fb00635fc caffe::SyncedMemory::mutable_cpu_data()
@ 0x7fafeec6a4 caffe::Blob<>::Reshape()
@ 0x7fafeecce4 caffe::Blob<>::Reshape()
@ 0x7faffd2af4 caffe::InputLayer<>::LayerSetUp()
@ 0x7fb00c9d28 caffe::Net<>::Init()
@ 0x7fb00cbf24 caffe::Net<>::Net()
@ 0x7fb00cf6dc FRCNN_API::Detector::Set_Model()
@ 0x4b7614 FRCNN_API::Detector::Detector()
@ 0x4b4318 main
@ 0x7faf3368a0 __libc_start_main
Aborted (core dumped)
I scoured the internet for a while and the closest relevant thread I found was this GitHub issue: https://github.com/NVIDIA/DIGITS/issues/1663. I tried out the
sudo ldconfig /usr/local/cuda/lib64
but it didn’t work. When I tried out another suggestion posted there:
sudo apt-get install nvidia-modprobe
, I get an error saying
Unable to locate package nvidia-modprobe
.
Can someone please help me as to how should I go about solving this problem? Thanks.
Hi,
The solution you mentioned is for x86 environment and cannot be used on Jetson.
Guess that you don’t use the correct GPU architecture to compile Caffe.
For example, please add TX2 architecture(sm=62) into Makefile.config:
https://github.com/BVLC/caffe/blob/master/Makefile.config.example#L37
CUDA_ARCH := -gencode arch=compute_20,code=sm_20 \
-gencode arch=compute_20,code=sm_21 \
-gencode arch=compute_30,code=sm_30 \
-gencode arch=compute_35,code=sm_35 \
-gencode arch=compute_50,code=sm_50 \
-gencode arch=compute_52,code=sm_52 \
-gencode arch=compute_60,code=sm_60 \
-gencode arch=compute_61,code=sm_61 \
<b>-gencode arch=compute_62,code=sm_62 \</b>
-gencode arch=compute_61,code=compute_61
Thanks.
Thanks for the help @AastaLLL, I modified Makefile.config file to add the TX2 architecture, recompiled it but I still get the same error.
Hi,
Please remember that it’s required to install OS and libraries from the same JetPack Installer.
The compatible CUDA driver for CUDA libraries is included in the OS image and cannot be found elsewhere.
We are checking this internally and will let you know the results later.
Thanks.
Hi,
Sorry, we didn’t notice that you are using ROS.
Could you check CUDA functionality with our official sample first?
/usr/local/cuda-8.0/bin/cuda-install-samples-8.0.sh .
cd NVIDIA_CUDA-8.0_Samples/0_Simple/vectorAdd
make
./vectorAdd
By the way, could you share the result you use JetPack3.0.
If possible, it’s recommended to use our latest JetPack3.2:
Thanks.
Hi, I ran the sample you provided. I got the following output:
[Vector addition of 50000 elements]
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
Copy output data from the CUDA device to the host memory
Test PASSED
Done
Could you please be more specific about which result should I share when using JetPack 3.0? The error I was getting is in the question itself.
Also, all of the libraries including the OS has been installed from the same JetPack (v3.0).
Thanks.
Hi,
Could you try to build Caffe with root authority?
sudo make
Thanks.
Hi rahulbohare1993,
Have you clarified the cause and resolved the error?
Any result can be shared?
Thanks