Libnvrtc.so.10.2 not found

Accelerated Computing Intelligent Video Analytics DeepStream SDK
#1

When I run 'deepstream-app -c deepstream_app_config_yoloV3.txt ', it gives following warning and errors.

(gst-plugin-scanner:3791): GStreamer-WARNING **: 10:10:37.244: Failed to load plugin ‘/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_infer.so’: /usr/local/cuda-10.1/lib64/libnvrtc.so.10.2: version `libnvrtc.so.10.2’ not found (required by /usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so.6)
** ERROR: <create_primary_gie_bin:107>: Failed to create ‘primary_gie_classifier’
** ERROR: <create_primary_gie_bin:161>: create_primary_gie_bin failed
** ERROR: <create_pipeline:1045>: create_pipeline failed
** ERROR: main:632: Failed to create pipeline
Quitting
App run failed

But my CUDA version is cuda-10.1. Why does it try to load a libnvrtc.so.10.2?

#2

Hi,

Thanks for your reporting.

We are trying to reproduce this issue.
Will update more information with you later.

#3

Hi,

We can run the sample without issue on a CUDA 10.1 host environment.

Could you check if the nvrtc library link to incorrect file on your environment?

$ ll /usr/local/cuda/lib64/libnvrtc*

lrwxrwxrwx 1 root root       25  二  26  2019 /usr/local/cuda/lib64/libnvrtc-builtins.so -> libnvrtc-builtins.so.10.1
lrwxrwxrwx 1 root root       29  二  26  2019 /usr/local/cuda/lib64/libnvrtc-builtins.so.10.1 -> libnvrtc-builtins.so.10.1.105
-rw-r--r-- 1 root root  4748560  二  26  2019 /usr/local/cuda/lib64/libnvrtc-builtins.so.10.1.105
lrwxrwxrwx 1 root root       16  二  26  2019 /usr/local/cuda/lib64/libnvrtc.so -> libnvrtc.so.10.1
lrwxrwxrwx 1 root root       20  二  26  2019 /usr/local/cuda/lib64/libnvrtc.so.10.1 -> libnvrtc.so.10.1.105
-rw-r--r-- 1 root root 21765416  二  26  2019 /usr/local/cuda/lib64/libnvrtc.so.10.1.105

Thanks.

#4

I run the same “deepstream-app -c deepstream_app_config_yoloV3.txt”

and also got this error

0:00:00.342183271 23906 0x556bdf88e950 ERROR nvinfer gstnvinfer.cpp:511:gst_nvinfer_logger:<primary_gie_classifier> NvDsInferContext[UID 1]:initialize(): Could not open custom lib: libnvrtc.so.10.2: cannot open shared object file: No such file or directory
0:00:00.342224331 23906 0x556bdf88e950 WARN nvinfer gstnvinfer.cpp:692:gst_nvinfer_start:<primary_gie_classifier> error: Failed to create NvDsInferContext instance
0:00:00.342231733 23906 0x556bdf88e950 WARN nvinfer gstnvinfer.cpp:692:gst_nvinfer_start:<primary_gie_classifier> error: Config file path: /opt/nvidia/deepstream/deepstream-4.0/sources/objectDetector_Yolo/config_infer_primary_yoloV3.txt, NvDsInfer Error: NVDSINFER_CUSTOM_LIB_FAILED
** ERROR: main:651: Failed to set pipeline to PAUSED
Quitting
ERROR from primary_gie_classifier: Failed to create NvDsInferContext instance
Debug info: gstnvinfer.cpp(692): gst_nvinfer_start (): /GstPipeline:pipeline/GstBin:primary_gie_bin/GstNvInfer:primary_gie_classifier:
Config file path: /opt/nvidia/deepstream/deepstream-4.0/sources/objectDetector_Yolo/config_infer_primary_yoloV3.txt, NvDsInfer Error: NVDSINFER_CUSTOM_LIB_FAILED
App run failed

My cuda version is 10.1
I have nvrtc library version 10.1 (/usr/local/cuda-10.1/lib64/libnvrtc.so.10.1)

#6

Hi,

Could you check libnvrtc librarary status with following command:

$ ll /usr/local/cuda/lib64/libnvrtc*

If the libnvrtc.so somehow link to libnvrtc.so.10.2, could you manually update it back to libnvrtc.so.10.1.
Thanks.

#7 
$ ll /usr/local/cuda/lib64/libnvrtc*    
lrwxrwxrwx 1 root root       25 Mar  6 03:07 /usr/local/cuda/lib64/libnvrtc-builtins.so -> libnvrtc-builtins.so.10.1*
lrwxrwxrwx 1 root root       29 Mar  6 03:07 /usr/local/cuda/lib64/libnvrtc-builtins.so.10.1 -> libnvrtc-builtins.so.10.1.105*
-rwxr-xr-x 1 root root  4748560 Mar  6 03:07 /usr/local/cuda/lib64/libnvrtc-builtins.so.10.1.105*
lrwxrwxrwx 1 root root       16 Mar  6 03:07 /usr/local/cuda/lib64/libnvrtc.so -> libnvrtc.so.10.1*
lrwxrwxrwx 1 root root       20 Mar  6 03:07 /usr/local/cuda/lib64/libnvrtc.so.10.1 -> libnvrtc.so.10.1.105*
-rwxr-xr-x 1 root root 21765416 Mar  6 03:07 /usr/local/cuda/lib64/libnvrtc.so.10.1.105*

It’s already linked to libnvrtc.so.10.1 .

#8

Hi,

Have same problem (using this guide):
docker container: nvidia/cuda:10.1-cudnn7-devel-ubuntu18.04

apt install ./deepstream-4.0_4.0.2-1_amd64.deb

Somehow it requires libnvrtc.so.10.2. But the following command points to cuda 10.1
ll /usr/local/cuda/lib64/libnvrtc*

And for deepstream 4.0.1 the same routine works just fine :)

Thank you)

#9

Hi,

Sorry for keeping you waiting.

It looks like the link issue comes from TensorRT package.
Could you share your TensorRT information with this command:

 dpkg -l | grep TensorRT

Please make sure the installed TensorRT is compiled with CUDA 10.1.

Thanks.

#10

I reinstalled everything and somehow managed to make it work by downgrading driver version 430.50