How to solve error == cudaSuccess (35 vs. 0) CUDA driver version is insufficient for CUDA runtime version?

Robotics & Edge Computing Jetson & Embedded Systems Jetson TX2
1

Hi,when trying to run SSD Caffe in tx2.

E0509 05:09:58.342988  8436 common.cpp:113] Cannot create Cublas handle. Cublas won't be available.
E0509 05:09:58.348474  8436 common.cpp:120] Cannot create Curand generator. Curand won't be available.
F0509 05:09:58.351529  8436 common.cpp:151] Check failed: error == cudaSuccess (35 vs. 0)  CUDA driver version is insufficient for CUDA runtime version

Another problem is when i make runtest in SSD caffe:

[ RUN      ] CPUBBoxUtilTest.TestOutputBBox
F0509 03:13:49.543218  5767 test_bbox_util.cpp:279] Check failed: out_bbox.xmax() == 50. (50 vs. 50) 
*** Check failure stack trace: ***
    @       0x7f94531718  google::LogMessage::Fail()
    @       0x7f94533614  google::LogMessage::SendToLog()
    @       0x7f94531290  google::LogMessage::Flush()
    @       0x7f94533eb4  google::LogMessageFatal::~LogMessageFatal()
    @           0xc57730  caffe::CPUBBoxUtilTest_TestOutputBBox_Test::TestBody()
    @           0xecff64  testing::internal::HandleExceptionsInMethodIfSupported<>()
    @           0xec8ebc  testing::Test::Run()
    @           0xec8ff8  testing::TestInfo::Run()
    @           0xec9104  testing::TestCase::Run()
    @           0xecab68  testing::internal::UnitTestImpl::RunAllTests()
    @           0xecae7c  testing::UnitTest::Run()
    @           0x92c230  main
    @       0x7f8f3788a0  __libc_start_main

Here is my Caffe configuration Summary

- ******************* Caffe Configuration Summary *******************
-- General:
--   Version           :   1.0.0-rc3
--   Git               :   ssdv1.0-dirty
--   System            :   Linux
--   C++ compiler      :   /usr/bin/c++
--   Release CXX flags :   -O3 -DNDEBUG -fPIC -Wall -Wno-sign-compare -Wno-uninitialized
--   Debug CXX flags   :   -g -fPIC -Wall -Wno-sign-compare -Wno-uninitialized
--   Build type        :   Release
-- 
--   BUILD_SHARED_LIBS :   ON
--   BUILD_python      :   ON
--   BUILD_matlab      :   OFF
--   BUILD_docs        :   ON
--   CPU_ONLY          :   OFF
--   USE_OPENCV        :   ON
--   USE_LEVELDB       :   ON
--   USE_LMDB          :   ON
--   ALLOW_LMDB_NOLOCK :   OFF
-- 
-- Dependencies:
--   BLAS              :   Yes (Atlas)
--   Boost             :   Yes (ver. 1.58)
--   glog              :   Yes
--   gflags            :   Yes
--   protobuf          :   Yes (ver. 3.5.1)
--   lmdb              :   Yes (ver. 0.9.17)
--   LevelDB           :   Yes (ver. 1.18)
--   Snappy            :   Yes (ver. 1.1.3)
--   OpenCV            :   Yes (ver. 3.4.0)
--   CUDA              :   Yes (ver. 9.0)
-- 
-- NVIDIA CUDA:
--   Target GPU(s)     :   Auto
--   GPU arch(s)       :   sm_62
--   cuDNN             :   Yes (ver. 7.0.5)
-- 
-- Python:
--   Interpreter       :   /usr/bin/python2.7 (ver. 2.7.12)
--   Libraries         :   /usr/lib/aarch64-linux-gnu/libpython2.7.so (ver 2.7.12)
--   NumPy             :   /usr/lib/python2.7/dist-packages/numpy/core/include (ver 1.11.0)
-- 
-- Documentaion:
--   Doxygen           :   No
--   config_file       :   
-- 
-- Install:
--   Install path      :   /root/caffe/install
--
2

Hi,

Could you share the step you setup the environment?

Usually, insufficient driver error is caused by the incompatible CUDA driver and CUDA library.
Please remember that the OS image and CUDA libraries needs to come from the same JetPack installer.

Thanks.

3

Actually,i setup the environment in docker and use Jetpack3.2 installer. Here is Dockerfile which i refer to :
[url]https://github.com/open-horizon/cogwerx-jetson-tx2/blob/master/Dockerfile.cudabase[/url].

4

Hi,

Not sure if this issue comes from docker setting.
There are some required configurations to enable GPU functionality in the docker container.

Here is a topic for docker setting for your reference:
[url]https://devtalk.nvidia.com/default/topic/1000224/jetson-tx2/docker-on-the-tx2[/url]

Thanks.