caffe2 inference model report :CUDA driver version is insufficient for CUDA runtime version

Accelerated Computing CUDA CUDA Programming and Performance
1

yesterday installed caffe2, try to run maksrcnn demo, the error “CUDA driver version is insufficient for CUDA runtime version” occurs, but that error never happen before use tensorflow. pls help me solve it! thanks a lot!

system information

GPU:1080Ti

CUDA Version 9.0.176

CUDNN_MAJOR 7

Driver Version: 384.130

OS Ubuntu 17.10

Python version:3.6.8

error detail

INFO net.py: 60: Loading weights from: /tmp/detectron-download-cache/35861858/12_2017_baselines/e2e_mask_rcnn_R-101-FPN_2x.yaml.02_32_51.SgT4y1cO/output/train/coco_2014_train:coco_2014_valminusminival/generalized_rcnn/model_final.pkl
INFO net.py: 96: conv1_w loaded from weights file into gpu_0/conv1_w: (64, 3, 7, 7)
Traceback (most recent call last):
File “tools/infer_simple.py”, line 185, in
main(args)
File “tools/infer_simple.py”, line 135, in main
model = infer_engine.initialize_model_from_cfg(args.weights)
File “/home/bruce/detectron/detectron/core/test_engine.py”, line 329, in initialize_model_from_cfg
model, weights_file, gpu_id=gpu_id,
File “/home/bruce/detectron/detectron/utils/net.py”, line 112, in initialize_gpu_from_weights_file
src_blobs[src_name].astype(np.float32, copy=False))
File “/home/bruce/anaconda3/envs/maskrcnn/lib/python3.6/site-packages/caffe2/python/workspace.py”, line 352, in FeedBlob
return _Workspace_feed_blob(ws, name, arr, device_option)
File “/home/bruce/anaconda3/envs/maskrcnn/lib/python3.6/site-packages/caffe2/python/workspace.py”, line 711, in _Workspace_feed_blob
return ws.create_blob(name).feed(arr, device_option)
File “/home/bruce/anaconda3/envs/maskrcnn/lib/python3.6/site-packages/caffe2/python/workspace.py”, line 741, in _Blob_feed
return blob._feed(arg, device_option)
RuntimeError: [enforce fail at common_gpu.cc:98] error == cudaSuccess. 35 vs 0. Error at: /opt/conda/conda-bld/pytorch-nightly_1560316055483/work/caffe2/core/common_gpu.cc:98: CUDA driver version is insufficient for CUDA runtime version
frame #0: c10::ThrowEnforceNotMet(char const*, int, char const*, std::string const&, void const*) + 0x59 (0x7fa791f4d8a9 in /home/bruce/anaconda3/envs/maskrcnn/lib/python3.6/site-packages/caffe2/python/…/…/torch/lib/libc10.so)
frame #1: caffe2::CaffeCudaGetDevice() + 0x8f6 (0x7fa746dabdc6 in /home/bruce/anaconda3/envs/maskrcnn/lib/python3.6/site-packages/caffe2/python/…/…/torch/lib/libcaffe2_gpu.so)
frame #2: + 0x2cf8165 (0x7fa7485f4165 in /home/bruce/anaconda3/envs/maskrcnn/lib/python3.6/site-packages/caffe2/python/…/…/torch/lib/libcaffe2_gpu.so)
frame #3: + 0x68eb7 (0x7fa79266feb7 in /home/bruce/anaconda3/envs/maskrcnn/lib/python3.6/site-packages/caffe2/python/caffe2_pybind11_state_gpu.cpython-36m-x86_64-linux-gnu.so)
frame #4: + 0x6a1ab (0x7fa7926711ab in /home/bruce/anaconda3/envs/maskrcnn/lib/python3.6/site-packages/caffe2/python/caffe2_pybind11_state_gpu.cpython-36m-x86_64-linux-gnu.so)
frame #5: + 0xf07b1 (0x7fa7926f77b1 in /home/bruce/anaconda3/envs/maskrcnn/lib/python3.6/site-packages/caffe2/python/caffe2_pybind11_state_gpu.cpython-36m-x86_64-linux-gnu.so)

2

Update your driver to the latest one for your GPU.

3

yeah , i try to upgrade my gup driver, but got the error as below, even i restart my computer the error still here. the package is NVIDIA-Linux-x86_64-430.26.run.

nvidia-installer log file ‘/var/log/nvidia-installer.log’
creation time: Thu Jun 13 18:19:54 2019
installer version: 430.26

PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin

nvidia-installer command line:
./nvidia-installer

Unable to load: nvidia-installer ncurses v6 user interface

Using: nvidia-installer ncurses user interface
→ Detected 12 CPUs online; setting concurrency level to 12.
ERROR: An NVIDIA kernel module ‘nvidia-drm’ appears to already be loaded in your kernel. This may be because it is in use (for example, by an X server, a CUDA program, or the NVIDIA Persistence Daemon), but this may also happen if your kernel was configured without support for module unloading. Please be sure to exit any programs that may be using the GPU(s) before attempting to upgrade your driver. If no GPU-based programs are running, you know that your kernel supports module unloading, and you still receive this message, then an error may have occured that has corrupted an NVIDIA kernel module’s usage count, for which the simplest remedy is to reboot your computer.

4

I guess you’ll need to read that ERROR message and take appropriate action.