DeepStream 6.0.1 Docker container missing dependencies?

I have a custom Dockerfile that runs a custom inference program (based on one of the DeepStream examples), and the Docker image is based off of nvcr.io/nvidia/deepstream-l4t:6.0.1-base:

FROM        nvcr.io/nvidia/deepstream-l4t:6.0.1-base
RUN         apt-get update && apt-get install ffmpeg -y
COPY        libgstinterpipe.so /usr/lib/aarch64-linux-gnu/gstreamer-1.0/
COPY        <copy my custom inference binary here>
COPY        <copy my custom detector here>
WORKDIR     <working directory here>
ENTRYPOINT [<entry point here to my inference binary>]

This same output image worked perfectly when it was based on nvcr.io/nvidia/deepstream-l4t:5.0.1-20.09-samples, and the only thing I changed was the base image version and I rebuilt the inference binary using DS6.0.1. However, now when I run a container off the 6.0.1 image, I get a number of missing dependencies.

The first one is: **/opt/nvidia/deepstream/deepstream-6.0/sources/apps/sample_apps/custom_inference/inference_binary: error while loading shared libraries: libcudart.so.10.2: cannot open shared object file: No such file or directory**

This prevents the container from spinning up successfully. I can get around this by mounting /usr/local/cuda/lib64 (where the libcudart.so.10.2 file is located on my Nano) to the container, but this doesn’t seem like the right solution? After I “fix” that issue, I get a bunch of “Failed to load plugin” errors:

**(gst-plugin-scanner:20): GStreamer-WARNING **: 03:55:32.165: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_inferaudio.so': libnvparsers.so.8: cannot open shared object file: No such file or directory**
**(gst-plugin-scanner:20): GStreamer-WARNING **: 03:55:32.180: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_inferserver.so': libtritonserver.so: cannot open shared object file: No such file or directory**
**(gst-plugin-scanner:20): GStreamer-WARNING **: 03:55:32.208: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_osd.so': libnvinfer.so.8: cannot open shared object file: No such file or directory**
**(gst-plugin-scanner:20): GStreamer-WARNING **: 03:55:32.220: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_udp.so': librivermax.so.0: cannot open shared object file: No such file or directory**
**(gst-plugin-scanner:20): GStreamer-WARNING **: 03:55:32.230: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_deepstream_bins.so': libnvparsers.so.8: cannot open shared object file: No such file or directory**
**(gst-plugin-scanner:20): GStreamer-WARNING **: 03:55:32.233: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_infer.so': libnvparsers.so.8: cannot open shared object file: No such file or directory**

I don’t actually know how those are used. But the real problem seems to come when I try to use nvinfer in my inference pipeline. It fails to create the GStreamer element and causes the whole pipeline to fail. Then, if I check gst-inspect-1.0 in my container, I don’t see any references to nvinfer at all.

Any ideas as to what is going on here? I thought the whole point of using a DeepStream Docker image was that it was supposed to have all of the required dependencies already?

In an act of desperation, I reflashed my Jetson Nano with a clean install of L4T 32.7.1 (Jetpack 4.6.1) and installed DeepStream 6.0.1. I rebuilt my inference binary, rebuilt my Docker image (using 6.0.1-samples)… and it is working!

Prior to the reflash, I had upgraded my Nano from L4T 32.4.4 (Jetpack 4.4.1) → L4T 32.7.1 (Jetpack 4.6.1) using minor release update process here. I updated from DeepStream 5.0 → DeepStream 6.0.1 using the tar package method here.

But obviously something didn’t work out right. Is that not the correct procedure to update a device?

I can find libcudart.so.10.2 under container nvcr.io/nvidia/deepstream-l4t:6.0.1-base
Did you compile your binary linked with cuda-10.2?

But the real problem seems to come when I try to use nvinfer in my inference pipeline. It fails to create the GStreamer element and causes the whole pipeline to fail. Then, if I check gst-inspect-1.0 in my container, I don’t see any references to nvinfer at all.

I also can get the output of gst-inspect-1.0 nvinfer in this container.

Prior to the reflash, I had upgraded my Nano from L4T 32.4.4 (Jetpack 4.4.1) → L4T 32.7.1 (Jetpack 4.6.1) using minor release update process here. I updated from DeepStream 5.0 → DeepStream 6.0.1 using the tar package method here.

But obviously something didn’t work out right. Is that not the correct procedure to update a device?

Can you specify more detailed?

It must not have been linked properly. How can I fix this?

Yes. What would you like to know? I have a Jetson Nano developer kit that I had originally installed Jetpack 4.4.1 and DeepStream 5.0 on. I used that to compile my inference binary, and built the Docker image with the Dockerfile shown above (but using [Preformatted text](http://nvcr.io/nvidia/deepstream-l4t:5.0.1-20.09-samples) base image.

My camera supplier issued a new camera driver that required Jetpack 4.6.1 and DeepStream 6.0.1, so I followed the instructions I found online (the ones I linked to above) to “upgrade” the device to the correct versions. I also uninstalled the DeepStream 5.0 package according to the instructions.

Then I updated my Makefile and Dockerfile to point to DS6.0, rebuilt the inference binary, and rebuilt the Docker image. Then when I tried to run the Docker container, I got the issues mentioned in my original post.

It must not have been linked properly. How can I fix this?
You can refer to any sample Makefile about how to link with cuda specified version library

Then I updated my Makefile and Dockerfile to point to DS6.0, rebuilt the inference binary, and rebuilt the Docker image. Then when I tried to run the Docker container, I got the issues mentioned in my original post.

Did you mean error you mentioned in the topic description?
Please follow this Quickstart Guide — DeepStream 6.0 Release documentation Install latest NVIDIA BSP packages
to update bsp packages and try again.

Ah ok… Yes, I checked my Makefile and I am linking to CUDA 10.2.

Am I to understand that installing a new version of Jetpack requires you to update the BSP packages? I can try this today. I have a second Jetson that needs to be updated to Jetpack 4.6.1 in order to run the new camera drivers.

So my process will be:

  1. Upgrade Jetpack using minor release update process
  2. Update BSP packages
  3. Update DeepStream using method 2 (tar package)

@Amycao Unfortunately, I ran into an error on my Jetpack upgrade (step 1 above):

Setting up nvidia-l4t-bootloader (32.7.2-20220420143418) ...
3448-300-0000--1--jetson-nano-devkit-
Starting bootloader post-install procedure.
ERROR. Procedure for bootloader update FAILED.
Cannot install package. Exiting...
dpkg: error processing package nvidia-l4t-bootloader (--configure):
 installed nvidia-l4t-bootloader package post-installation script subprocess returned error exit status 1
Setting up nvidia-l4t-xusb-firmware (32.7.2-20220420143418) ...
3448-300-0000--1--jetson-nano-devkit-
Starting xusb firmware post-install procedure.
ERROR. Procedure for xusb firmware update FAILED.
Cannot install package. Exiting...
dpkg: error processing package nvidia-l4t-xusb-firmware (--configure):
 installed nvidia-l4t-xusb-firmware package post-installation script subprocess returned error exit status 1
dpkg: dependency problems prevent configuration of nvidia-l4t-initrd:
 nvidia-l4t-initrd depends on nvidia-l4t-xusb-firmware (= 32.7.2-20220420143418); however:
  Package nvidia-l4t-xusb-firmware is not configured yet.

dpkg: error processing package nvidia-l4t-initrd (--configure):
 dependency problems - leaving unconfigured
No apport report written because the error message indicates its a followup error from a previous failure.
                                                                                                          Setting up cuda-documentation-10-2 (10.2.300-1) ...
Setting up cuda-toolkit-10-2 (10.2.460-1) ...
Setting alternatives
update-alternatives: using /usr/local/cuda-10.2 to provide /usr/local/cuda (cuda) in auto mode
update-alternatives: using /usr/local/cuda-10.2 to provide /usr/local/cuda-10 (cuda-10) in auto mode
Errors were encountered while processing:
 nvidia-l4t-bootloader
 nvidia-l4t-xusb-firmware
 nvidia-l4t-initrd
E: Sub-process /usr/bin/dpkg returned an error code (1)

Searching for that those errors leads me to threads like these, which suggest a reflash:

Yes.

Did you try sudo apt-get dist-upgrade if it fixed or not? if not, suggest you reflash the device.

Won’t sudo apt-get dist-upgrade upgrade me to the latest available Jetpack? I need to be on 4.6.1 for my camera drivers to work.

It will upgrade according to settings in /etc/apt/sources.list.d/nvidia-l4t-apt-source.list

There is no update from you for a period, assuming this is not an issue anymore.
Hence we are closing this topic. If need further support, please open a new one.
Thanks

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.