cudnn_conv_layer.cpp:53] Check failed: status == CUDNN_STATUS_SUCCESS (4 vs. 0) CUDNN_STATUS_INTERNAL_ERROR

Autonomous Machines Jetson & Embedded Systems Jetson TX2
1

hi I meet this problem on tx2 when I am trying to run caffe-ssd.

After building caffe-ssd I can run it with :
python ssd_detect_py

but ,when I try to use ssd_detect.bin,it just list the following errors:

F1216 08:25:32.651688  1943 cudnn_conv_layer.cpp:53] Check failed: status == CUDNN_STATUS_SUCCESS (4 vs. 0)  CUDNN_STATUS_INTERNAL_ERROR
*** Check failure stack trace: ***
    @       0x7fb3201718  google::LogMessage::Fail()
    @       0x7fb3203614  google::LogMessage::SendToLog()
    @       0x7fb3201290  google::LogMessage::Flush()
    @       0x7fb3203eb4  google::LogMessageFatal::~LogMessageFatal()
    @       0x7fb35d43e8  caffe::CuDNNConvolutionLayer<>::LayerSetUp()
    @       0x7fb363fbc4  caffe::Net<>::Init()
    @       0x7fb364193c  caffe::Net<>::Net()
    @           0x40870c  Detector::Detector()
    @           0x4053ec  main
    @       0x7fb27d78a0  __libc_start_main
Aborted (core dumped)

I have cancled USE_CUDNN := 1 in the Makefile.config before build this project.

Can you give me some advice?Thanks.

2

Hi,

We have checked caffe-ssd functionality before, and it can work correctly on TX1.
[url]Compiling SSD (Single Shot Detection) on Jetson TX1 - Jetson TX1 - NVIDIA Developer Forums

CUDNN_STATUS_INTERNAL_ERROR is usually caused by incorrect driver/libraries version.
From the error message, your application tries to call cuDNN implementation but has been turned off.

Please turn on the cuDNN flag, clean build the caffe-ssd branch and try it again.
Thanks.

3

hi,I hace fix this problem,this problem is caused by failed install cuDNN,So I reinstall it and make.

4

hello, I had same error … I use nivida-384 cuda8 and cudnn5.1 .
can u tell me what version you choose? thank you

5

Hi,

Do you try to run caffe-ssd on a desktop GPU?
If you are using this branch, it should work well on CUDA 8.0 and cuDNN5.1.

Thanks.

6

Hello everyone. I’m trying to run caffe/models/bvlc_googlenet with my own data.
when I run: nvidia@tegra-ubuntu:~/caffe/models/bvlc_googlenet$ sudo /home/nvidia/caffe/tools/caffe train -solver solver.prototxt

I get the next error:

F1208 19:33:29.198784 32130 cudnn_conv_layer.cpp:53] Check failed: status == CUDNN_STATUS_SUCCESS (4 vs. 0) CUDNN_STATUS_INTERNAL_ERROR
*** Check failure stack trace: ***
@ 0x7fa7803718 google::LogMessage::Fail()
@ 0x7fa7805614 google::LogMessage::SendToLog()
@ 0x7fa7803290 google::LogMessage::Flush()
@ 0x7fa7805eb4 google::LogMessageFatal::~LogMessageFatal()
@ 0x7fa7ac4098 caffe::CuDNNConvolutionLayer<>::LayerSetUp()
@ 0x7fa79c65ac caffe::Net<>::Init()
@ 0x7fa79c7dec caffe::Net<>::Net()
@ 0x7fa79d9b50 caffe::Solver<>::InitTestNets()
@ 0x7fa79da30c caffe::Solver<>::Init()
@ 0x7fa79da5b4 caffe::Solver<>::Solver()
@ 0x7fa7b51d30 caffe::Creator_SGDSolver<>()
@ 0x40b214 train()
@ 0x408888 main
@ 0x7fa733a8a0 __libc_start_main
Aborted (core dumped)

I’m using JETSON TX2 jetpack 3.0 with CUDA 8.0.62 cuDNN 5.1.10
Please I need your help because I’m beginner in ConvNN.

Thank you so much

7

Hi,

May I know how do you build the Caffe package?
It’s recommended to use jetsonhacks’ script to build it from source:
[url]https://github.com/jetsonhacks/installCaffeJTX2[/url]

Thanks.