Nvinfer not found on x86-64

[mdegans]

Please provide complete information as applicable to your setup.
• Hardware Platform (Jetson / GPU)
DGPU
• DeepStream Version
5.0.0
• TensorRT Version
it uninstalled itself
• NVIDIA GPU Driver Version (valid for GPU only)
440.95.01

After a recent apt-upgrade on x86-64, a bunch of cuda 10 stuff uninstalled itself in favor of cuda 11 things. At around the same time, DeepStream stopped working. I’ve tried re-installing from .deb but it appears the dependencies are not set correctly since otherwise it would install the required libraries along with it. My Nvidia apt source are as follows:

deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 /
deb http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 /

Nvinfer is installed, but gstreamer will no longer load the plugin. There were some errors about missing libraries which went away after a ldconfig (apparently the postinst script does not do this?), but it still wont’ load:

[mdegans@konjin] -- [/usr/lib/pkgconfig] 
 $ GST_DEBUG=4 gst-inspect-1.0 nvinfer
0:00:00.000155977 16224 0x562a31ab3a00 INFO                GST_INIT gst.c:586:init_pre: Initializing GStreamer Core Library version 1.14.5
0:00:00.000237226 16224 0x562a31ab3a00 INFO                GST_INIT gst.c:587:init_pre: Using library installed in /usr/lib/x86_64-linux-gnu
0:00:00.000292297 16224 0x562a31ab3a00 INFO                GST_INIT gst.c:607:init_pre: Linux konjin 4.15.0-111-generic #112-Ubuntu SMP Thu Jul 9 20:32:34 UTC 2020 x86_64
0:00:00.000799747 16224 0x562a31ab3a00 INFO                GST_INIT gstmessage.c:127:_priv_gst_message_initialize: init messages
0:00:00.001274844 16224 0x562a31ab3a00 INFO                GST_INIT gstcontext.c:84:_priv_gst_context_initialize: init contexts
0:00:00.001396651 16224 0x562a31ab3a00 INFO      GST_PLUGIN_LOADING gstplugin.c:317:_priv_gst_plugin_initialize: registering 0 static plugins
0:00:00.001442692 16224 0x562a31ab3a00 INFO      GST_PLUGIN_LOADING gstplugin.c:225:gst_plugin_register_static: registered static plugin "staticelements"
0:00:00.001450120 16224 0x562a31ab3a00 INFO      GST_PLUGIN_LOADING gstplugin.c:227:gst_plugin_register_static: added static plugin "staticelements", result: 1
0:00:00.001463824 16224 0x562a31ab3a00 INFO            GST_REGISTRY gstregistry.c:1727:ensure_current_registry: reading registry cache: /home/mdegans/.cache/gstreamer-1.0/registry.x86_64.bin
0:00:00.008433491 16224 0x562a31ab3a00 INFO            GST_REGISTRY gstregistrybinary.c:621:priv_gst_registry_binary_read_cache: loaded /home/mdegans/.cache/gstreamer-1.0/registry.x86_64.bin in 0.006955 seconds
0:00:00.008458050 16224 0x562a31ab3a00 INFO            GST_REGISTRY gstregistry.c:1583:scan_and_update_registry: Validating plugins from registry cache: /home/mdegans/.cache/gstreamer-1.0/registry.x86_64.bin
0:00:00.009136235 16224 0x562a31ab3a00 INFO            GST_REGISTRY gstregistry.c:1685:scan_and_update_registry: Registry cache has not changed
0:00:00.009143597 16224 0x562a31ab3a00 INFO            GST_REGISTRY gstregistry.c:1762:ensure_current_registry: registry reading and updating done, result = 1
0:00:00.009150778 16224 0x562a31ab3a00 INFO                GST_INIT gst.c:807:init_post: GLib runtime version: 2.56.4
0:00:00.009156113 16224 0x562a31ab3a00 INFO                GST_INIT gst.c:809:init_post: GLib headers version: 2.56.4
0:00:00.009159794 16224 0x562a31ab3a00 INFO                GST_INIT gst.c:810:init_post: initialized GStreamer successfully
No such element or plugin 'nvinfer'
[mdegans@konjin] -- [/usr/lib/pkgconfig] 
 $ GST_DEBUG=4 gst-inspect-1.0 nvstreammux
0:00:00.000160102 16226 0x556859afca00 INFO                GST_INIT gst.c:586:init_pre: Initializing GStreamer Core Library version 1.14.5
0:00:00.000232560 16226 0x556859afca00 INFO                GST_INIT gst.c:587:init_pre: Using library installed in /usr/lib/x86_64-linux-gnu
0:00:00.000269538 16226 0x556859afca00 INFO                GST_INIT gst.c:607:init_pre: Linux konjin 4.15.0-111-generic #112-Ubuntu SMP Thu Jul 9 20:32:34 UTC 2020 x86_64
0:00:00.000772692 16226 0x556859afca00 INFO                GST_INIT gstmessage.c:127:_priv_gst_message_initialize: init messages
0:00:00.001480638 16226 0x556859afca00 INFO                GST_INIT gstcontext.c:84:_priv_gst_context_initialize: init contexts
0:00:00.001603388 16226 0x556859afca00 INFO      GST_PLUGIN_LOADING gstplugin.c:317:_priv_gst_plugin_initialize: registering 0 static plugins
0:00:00.001643304 16226 0x556859afca00 INFO      GST_PLUGIN_LOADING gstplugin.c:225:gst_plugin_register_static: registered static plugin "staticelements"
0:00:00.001649351 16226 0x556859afca00 INFO      GST_PLUGIN_LOADING gstplugin.c:227:gst_plugin_register_static: added static plugin "staticelements", result: 1
0:00:00.001659764 16226 0x556859afca00 INFO            GST_REGISTRY gstregistry.c:1727:ensure_current_registry: reading registry cache: /home/mdegans/.cache/gstreamer-1.0/registry.x86_64.bin
0:00:00.008549931 16226 0x556859afca00 INFO            GST_REGISTRY gstregistrybinary.c:621:priv_gst_registry_binary_read_cache: loaded /home/mdegans/.cache/gstreamer-1.0/registry.x86_64.bin in 0.006876 seconds
0:00:00.008573039 16226 0x556859afca00 INFO            GST_REGISTRY gstregistry.c:1583:scan_and_update_registry: Validating plugins from registry cache: /home/mdegans/.cache/gstreamer-1.0/registry.x86_64.bin
0:00:00.009247802 16226 0x556859afca00 INFO            GST_REGISTRY gstregistry.c:1685:scan_and_update_registry: Registry cache has not changed
0:00:00.009254370 16226 0x556859afca00 INFO            GST_REGISTRY gstregistry.c:1762:ensure_current_registry: registry reading and updating done, result = 1
0:00:00.009258606 16226 0x556859afca00 INFO                GST_INIT gst.c:807:init_post: GLib runtime version: 2.56.4
0:00:00.009262435 16226 0x556859afca00 INFO                GST_INIT gst.c:809:init_post: GLib headers version: 2.56.4
0:00:00.009265105 16226 0x556859afca00 INFO                GST_INIT gst.c:810:init_post: initialized GStreamer successfully
0:00:00.029278693 16226 0x556859afca00 INFO      GST_PLUGIN_LOADING gstplugin.c:901:_priv_gst_plugin_load_file_for_registry: plugin "/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_multistream.so" loaded
0:00:00.029292132 16226 0x556859afca00 INFO     GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element "nvstreammux"
0:00:00.029397351 16226 0x556859afca00 INFO        GST_ELEMENT_PADS gstelement.c:670:gst_element_add_pad:<GstNvStreamMux@0x556859dda140> adding pad 'src'
Factory Details:
  Rank                     primary (256)
  Long-name                Stream multiplexer
  Klass                    Generic
  Description              N-to-1 pipe stream multiplexing
  Author                   NVIDIA Corporation. Post on Deepstream for Tesla forum for any queries @ https://devtalk.nvidia.com/default/board/209/

Plugin Details:
  Name                     nvdsgst_multistream
  Description              NVIDIA Multistream mux/demux plugin
  Filename                 /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_multistream.so
  Version                  5.0.0
  License                  Proprietary
  Source module            nvmultistream
  Binary package           NVIDIA Multistream Plugins
  Origin URL               http://nvidia.com/

GObject
 +----GInitiallyUnowned
       +----GstObject
             +----GstElement
                   +----GstNvStreamMux

Pad Templates:
  SINK template: 'sink_%u'
    Availability: On request
    Capabilities:
      video/x-raw(memory:NVMM)
                 format: { (string)NV12, (string)RGBA }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
      video/x-raw
                 format: { (string)NV12, (string)RGBA }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
  
  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-raw(memory:NVMM)
                 format: { (string)NV12, (string)RGBA }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]

Element has no clocking capabilities.
Element has no URI handling capabilities.

Pads:
  SRC: 'src'
    Pad Template: 'src'

Element Properties:
  name                : The name of the object
                        flags: readable, writable
                        String. Default: "nvstreammux0"
  parent              : The parent of the object
                        flags: readable, writable
                        Object of type "GstObject"
  batch-size          : Maximum number of buffers in a batch
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 1024 Default: 0 
  batched-push-timeout: Timeout in microseconds to wait after the first buffer is available
                        to push the batch even if the complete batch is not formed.
                        Set to -1 to wait infinitely
                        flags: readable, writable
                        Integer. Range: -1 - 2147483647 Default: -1 
  width               : Width of each frame in output batched buffer. This property MUST be set.
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 4294967295 Default: 0 
  height              : Height of each frame in output batched buffer. This property MUST be set.
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 4294967295 Default: 0 
  enable-padding      : Maintain input aspect ratio when scaling by padding with black bands.
                        flags: readable, writable
                        Boolean. Default: false
  gpu-id              : Set GPU Device ID
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 4294967295 Default: 0 
  live-source         : Boolean property to inform muxer that sources are live.
                        flags: readable, writable
                        Boolean. Default: false
  num-surfaces-per-frame: Max number of surfaces per frame
                        flags: readable, writable
                        Unsigned Integer. Range: 1 - 4 Default: 1 
  nvbuf-memory-type   : Type of NvBufSurface Memory to be allocated for output buffers
                        flags: readable, writable, changeable only in NULL or READY state
                        Enum "GstNvBufMemoryType" Default: 3, "nvbuf-mem-cuda-unified"
                           (0): nvbuf-mem-default - Default memory allocated, specific to particular platform
                           (1): nvbuf-mem-cuda-pinned - Allocate Pinned/Host cuda memory
                           (2): nvbuf-mem-cuda-device - Allocate Device cuda memory
                           (3): nvbuf-mem-cuda-unified - Allocate Unified cuda memory
  buffer-pool-size    : Maximum number of buffers in muxer's internal pool
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 1024 Default: 4 
  attach-sys-ts       : If set to TRUE, system timestamp will be attached as ntp timestamp.
                        If set to FALSE, ntp timestamp from rtspsrc, if available, will be attached.
                        flags: readable, writable
                        Boolean. Default: true
0:00:00.030233413 16226 0x556859afca00 INFO         GST_REFCOUNTING gstelement.c:3251:gst_element_dispose:<nvstreammux0> 0x556859dda140 dispose
0:00:00.030238916 16226 0x556859afca00 INFO        GST_ELEMENT_PADS gstelement.c:789:gst_element_remove_pad:<nvstreammux0> removing pad 'src'
0:00:00.030253743 16226 0x556859afca00 INFO         GST_REFCOUNTING gstelement.c:3297:gst_element_dispose:<nvstreammux0> 0x556859dda140 parent class dispose
0:00:00.030258418 16226 0x556859afca00 INFO         GST_REFCOUNTING gstelement.c:3328:gst_element_finalize:<nvstreammux0> 0x556859dda140 finalize
0:00:00.030262512 16226 0x556859afca00 INFO         GST_REFCOUNTING gstelement.c:3334:gst_element_finalize:<nvstreammux0> 0x556859dda140 finalize parent

Strangely, nvstreammux will load.

Any ideas, or a list of dependencies to install manually would be very helpful. Better yet, putting deepstream in the online apt repositories and fixing the Requires so they’re installed automatically would be very much appreciated.

Thanks

[mchi]

Do you have below file that is for nvinfer GST plugin?

/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_infer.so

[mdegans]

@mchi

[mdegans@konjin] -- [/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream] 
 $ file libnvdsgst_infer.so 
libnvdsgst_infer.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=d896e48e8e72d378e8b02601a524a8bc5ec7594a, not stripped
[mdegans@konjin] -- [/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream] 
 $ gst-inspect-1.0 libnvdsgst_infer.so 
Plugin Details:
  Name                     libnvdsgst_infer.so
  Description              Plugin for blacklisted file
  Filename                 /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_infer.so
  Version                  0.0.0
  License                  BLACKLIST
  Source module            BLACKLIST
  Binary package           BLACKLIST
  Origin URL               BLACKLIST


  0 features:

My understanding is (from this happening to my own plugins) is that “BLACKLIST” can mean a library the plugin depends on can’t be found, so i checked…

 $ ldd libnvdsgst_infer.so 
	linux-vdso.so.1 (0x00007ffdcd93b000)
	libcudart.so.10.2 => /usr/local/cuda-10.2/lib64/libcudart.so.10.2 (0x00007f325bd1a000)
	libnvds_infer.so => ///opt/nvidia/deepstream/deepstream-
...
	libcublas.so.11 => /usr/local/cuda-11.0/targets/x86_64-linux/lib/libcublas.so.11 (0x00007f3233f37000)
	libcudnn.so.8 => /usr/lib/x86_64-linux-gnu/libcudnn.so.8 (0x00007f3233d08000)
	libmyelin.so.1 => /usr/lib/x86_64-linux-gnu/libmyelin.so.1 (0x00007f32334e1000)
	libnvrtc.so.11.0 => /usr/local/cuda-11.0/targets/x86_64-linux/lib/libnvrtc.so.11.0 (0x00007f3231cf6000)
	libcublasLt.so.11 => /usr/local/cuda-11.0/targets/x86_64-linux/lib/libcublasLt.so.11 (0x00007f322729b000)

Not really an expert level C programmer here, but is it normal for a target to be linked to be linked against multiple versions of a library? Fwiw, all those libraries do exist on my system.

Additional info about my system’s current state here:

Error getting capabilities for device '/dev/nvhost-msenc'

@mchi Nope, but the deepstream docker image works, so i’m pretty sure it’s not driver related. My repositories are: [user@host] -- [/etc/apt/sources.list.d] $ cat cuda.list nvidia-docker.list nvidia-machine-learning.list deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 / deb https://nvidia.github.io/libnvidia-container/ubuntu18.04/$(ARCH) / deb https://nvidia.github.io/nvidia-container-runtime/ubuntu18.04/$(ARCH) / deb https://nvidia.github.io/nvidia-docker/ubu…

I’ve attached a full list of packages installed on this system (output of apt list --installed)

apt-list.txt (103.3 KB)

cuda packages:

cuda-cudart-10-1/unknown,now 10.1.243-1 amd64 [installed]
cuda-cudart-10-2/unknown,now 10.2.89-1 amd64 [installed,automatic]
cuda-cudart-11-0/unknown,now 11.0.194-1 amd64 [installed,automatic]
cuda-cudart-dev-10-2/unknown,now 10.2.89-1 amd64 [installed,automatic]
cuda-cudart-dev-11-0/unknown,now 11.0.194-1 amd64 [installed,automatic]
cuda-cufft-10-1/unknown,now 10.1.243-1 amd64 [installed]
cuda-cufft-10-2/unknown,now 10.2.89-1 amd64 [installed]
cuda-curand-10-2/unknown,now 10.2.89-1 amd64 [installed,automatic]
cuda-cusolver-10-2/unknown,now 10.2.89-1 amd64 [installed,automatic]
cuda-cusparse-10-2/unknown,now 10.2.89-1 amd64 [installed,automatic]
cuda-driver-dev-10-2/unknown,now 10.2.89-1 amd64 [installed,automatic]
cuda-driver-dev-11-0/unknown,now 11.0.194-1 amd64 [installed,automatic]
cuda-libraries-10-2/unknown,now 10.2.89-1 amd64 [installed]
cuda-libraries-11-0/unknown,now 11.0.2-1 amd64 [installed]
cuda-license-10-1/unknown,now 10.1.243-1 amd64 [installed,automatic]
cuda-license-10-2/unknown,now 10.2.89-1 amd64 [installed,automatic]
cuda-misc-headers-10-2/unknown,now 10.2.89-1 amd64 [installed,automatic]
cuda-npp-10-2/unknown,now 10.2.89-1 amd64 [installed,automatic]
cuda-npp-dev-10-2/unknown,now 10.2.89-1 amd64 [installed,automatic]
cuda-nvcc-10-2/unknown,now 10.2.89-1 amd64 [installed]
cuda-nvgraph-10-2/unknown,now 10.2.89-1 amd64 [installed,automatic]
cuda-nvjpeg-10-2/unknown,now 10.2.89-1 amd64 [installed,automatic]
cuda-nvrtc-10-2/unknown,now 10.2.89-1 amd64 [installed,automatic]
cuda-nvrtc-11-0/unknown,now 11.0.194-1 amd64 [installed,automatic]
cuda-repo-ubuntu1804/unknown,now 10.2.89-1 amd64 [installed]

[mchi]

@mdegans

	libcublas.so.11 => /usr/local/cuda-11.0/targets/x86_64-linux/lib/libcublas.so.11 (0x00007f3233f37000)
	libcudnn.so.8 => /usr/lib/x86_64-linux-gnu/libcudnn.so.8 (0x00007f3233d08000)
	libmyelin.so.1 => /usr/lib/x86_64-linux-gnu/libmyelin.so.1 (0x00007f32334e1000)
	libnvrtc.so.11.0 => /usr/local/cuda-11.0/targets/x86_64-linux/lib/libnvrtc.so.11.0 (0x00007f3231cf6000)
	libcublasLt.so.11 => /usr/local/cuda-11.0/targets/x86_64-linux/lib/libcublasLt.so.11 (0x00007f322729b000)

Hi @mdegans,
Sorry for delay! From above log, looks you installed CUDA 11.0 as default CUDA, i.e. /usr/local/cuda links to /usr/local/cuda-11.0, and DS5.0 does not support CUDA yet, so this may be the reason of the failure.
Could you re-link the /usr/local/cuda to /usr/local/cuda-10.2 and check again?

Thanks!

[mdegans]

@mchi

Making the suggested link fixed it. I don’t understand how that relates to ldd and the plugin linkage itself (the output of that command remainns the same), but it works now, so i’m not complaining. Thanks!