Hi, im trying to follow this to create a custom docker image using the l4t-base image, and then installing cuda, cudnn, tensorrt and opencv manually. When I run my program it fails with this error:
error while loading shared libraries: libnvdla_compiler.so: cannot open shared object file: No such file or directory
$ ldd myfile | grep nvdla
libnvdla_compiler.so => not found
libnvdla_runtime.so => /usr/lib/aarch64-linux-gnu/nvidia/libnvdla_runtime.so (0x0000ffff9d2f0000)
From other forum-posts I can see that I am missing the /usr/lib/aarch64-linux-gnu/tegra/libnvdla_compiler.so on my host system. However, if I run the exact same setup with the l4t-jetpack image as base, and still dont have the libnvdla_compiler.so anywhere on my host, the file suddenly appears in the docker container:
As we do not need all the packages from Jetpack, we would like to only install the necessary packages to keep the docker image as small as possible. Is it something I’m missing? I have all the nvidia-container packages
I found that the libnvdla_compiler.so file is not in Linux_for_Tegra/nv_tegra/nvidia_drivers.tbz2 in Jetpack 6, but present in Jetpack 5. If I simply copy the file from the BSP driver package for version 35.4.1 to my 36.2 jetson host, my binary file runs fine, although not the most graceful solution…
I wonder however if it’s correct that a “compiler” package should have been linked to my binary in the first place. I would assume that I only need the runtime file?
Hi @hvbotten, I’m not certain about all the intricacies of this regarding the source package of the nvdla drivers, however I’ve migrated to a similar approach over at jetson-containers installing CUDA/cuDNN/TensorRT/ect from debians. On JetPack 6, I’m using ubuntu:2204 as base. Here is what I’ve found:
libnvdla_runtime.so still gets mounted into the container from the host device by /etc/nvidia-container-runtime/host-files-for-container.d/drivers.csv when --runtime nvidia is used
I had to set the following environment variables in my cuda dockerfile to get these device/driver mounts working by the NVIDIA container runtime and the GPU recognized:
Interesting, I’m currently just getting things to work so I’ll stick with the jetpack 6 base image for now since it’s working, though i will try this after the holidays!
Why are you using ubuntu22.04 base image, and not the l4t-base image? Is it unnecessary?
When I tried installing the nvidia-cuda on the l4t-base image I got a lot of errors like this cuda-compat-12-2 : PreDepends: nvidia-l4t-core but it is not installable . Is that why I see you have the ar x /var/cuda-tegra-repo-*/cuda-compat-*.deb line etc… in your Dockerfile?
The last attempt I made I tried to only install the most necessary package, like this:
Are there any benefits of installing it the way you do over the sample l4t dockerfile?
I am trying to create a deployment runtime docker image as small as possible so I dont need the dev packages, but if your image ends up considerably smaller that would be nice. With only cuda-toolkit and libnvinfer8 my image gets to ~10GB. I’m a little unsure whether TensorRT requires cudnn8 for infering or not, haven’t tested yet.
I also had some issues with non-l4t-base docker images, specifically with Gstreamer. I tried installing Gstreamer in a docker container on both a l4t-jetpack image and a l4t-base image. On the l4t-jetpack image I get a bunch of warnings, which makes me think that it wasn’t installed correctly (maybe because of some l4t dependencies?). But on the l4t-base image it works without any warnings, and I installed it the same way on both docker files (just different base images). Is it correct that the l4t-jetpack image is not using the l4t-base image (but maybe just ubuntu22.04)? I see that on l4t-jetpack none of the nvidia-cuda/opencv packages are available, and no nvidia-l4t-* packages are installed.
Here are the gstreamer warnings from the l4t-jetpack image. I still get images from the gstreamer camera though, despite the warnings…
(gst-plugin-scanner:97): GStreamer-WARNING **: 14:32:33.312: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnveglstreamsrc.so': libnvbufsurface.so.1.0.0: cannot open shared object file: No such file or directory
(gst-plugin-scanner:97): GStreamer-WARNING **: 14:32:33.313: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvvideo4linux2.so': libnvbufsurface.so.1.0.0: cannot open shared object file: No such file or directory
(gst-plugin-scanner:97): GStreamer-WARNING **: 14:32:33.314: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvvidconv.so': libnvbufsurface.so.1.0.0: cannot open shared object file: No such file or directory
(gst-plugin-scanner:97): GStreamer-WARNING **: 14:32:33.318: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvv4l2camerasrc.so': libnvbufsurface.so.1.0.0: cannot open shared object file: No such file or directory
(gst-plugin-scanner:97): GStreamer-WARNING **: 14:32:33.321: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvarguscamerasrc.so': libnvargus_socketclient.so: cannot open shared object file: No such file or directory
(gst-plugin-scanner:97): GStreamer-WARNING **: 14:32:33.322: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvvideosinks.so': libnvbufsurface.so.1.0.0: cannot open shared object file: No such file or directory
(gst-plugin-scanner:97): GStreamer-WARNING **: 14:32:33.323: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvivafilter.so': libnvbufsurftransform.so.1.0.0: cannot open shared object file: No such file or directory
(gst-plugin-scanner:97): GStreamer-WARNING **: 14:32:33.324: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvjpeg.so': libnvjpeg.so: cannot open shared object file: No such file or directory
(gst-plugin-scanner:97): GStreamer-WARNING **: 14:32:33.324: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvcompositor.so': libnvbufsurface.so.1.0.0: cannot open shared object file: No such file or directory
and this:
$ gst-inspect-1.0 --version
No EGL Display
nvbufsurftransform: Could not get EGL display connection
(Argus) Error FileOperationFailed: Connecting to nvargus-daemon failed: No such file or directory (in src/rpc/socket/client/SocketClientDispatch.cpp, function openSocketConnection(), line 204)
(Argus) Error FileOperationFailed: Cannot create camera provider (in src/rpc/socket/client/SocketClientDispatch.cpp, function createCameraProvider(), line 106)
Adapting that l4t-jetpack dockerfile to suit your needs may be easier for a one-off application - my approach was for being able to change CUDA versions in the future and integrate with the rest of the AI/ML container stack from jetson-containers (which can get quite complex and have many variants, which is why that project is modularized for builds and testing)
I believe that l4t-jetpack is still using l4t-base as its base image.
GStreamer and those NVIDIA plugins should already be present in l4t-jetpack - the plugins get automatically mounted when --runtime nvidia is used (see /etc/nvidia-container-runtime/host-files-for-container.d/drivers.csv)
You need to add --volume /tmp/argus_socket:/tmp/argus_socket to your docker run command for Argus to work inside the container. There are additional mounts you can add for enabling display inside container if desired.
Weird, now (without installing anything) the /usr/lib/aarch64-linux-gnu/nvidia/libnvdla_compiler.so suddenly exists. This is slightly confusing.
@dusty_nv How would you install GStreamer if not using the l4t-jetpack base image? Right now we tried doing this (but got the errors/warnings as explained earlier):
Also regarding the Argus warnings, as far as I know we do not use anything called Argus. Those errors just appeared (sometimes, not always) after running gst-inspect-1.0 --version on the l4t-jetpack image
libnvdla_compiler.so is specified to automatically be mounted by /etc/nvidia-container-runtime/host-files-for-container.d/drivers.csv, so it should appear when --runtime nvidia is used to start container.
This is how I install GStreamer, and it works against ubuntu:22.04 base image:
I have always started the container with --runtime nvidia which is why I was confused that it sometimes doesn’t appear inside the docker container. libnvdla_compiler.so is also not present on my host, only inside the container (sometimes).
I tried pulling your gstreamer image and I still get the warnings. Unless there is something obvious I am missing I’ll just ignore these warnings for now, as the camera is still able to capture images. Maybe it has something to do with the way I flashed the Jetson
$ docker run --runtime nvidia --rm dustynv/gstreamer:r36.2.0 gst-inspect-1.0 --version
(gst-plugin-scanner:19): GLib-GObject-WARNING **: 10:07:49.012: cannot register existing type 'GstRtpSrc'
(gst-plugin-scanner:19): GLib-GObject-CRITICAL **: 10:07:49.012: g_type_add_interface_static: assertion 'G_TYPE_IS_INSTANTIATABLE (instance_type)' failed
(gst-plugin-scanner:19): GLib-CRITICAL **: 10:07:49.012: g_once_init_leave: assertion 'result != 0' failed
(gst-plugin-scanner:19): GStreamer-CRITICAL **: 10:07:49.012: gst_element_register: assertion 'g_type_is_a (type, GST_TYPE_ELEMENT)' failed
(gst-plugin-scanner:19): GLib-GObject-WARNING **: 10:07:49.012: cannot register existing type 'GstRtpSink'
(gst-plugin-scanner:19): GLib-GObject-CRITICAL **: 10:07:49.012: g_type_add_interface_static: assertion 'G_TYPE_IS_INSTANTIATABLE (instance_type)' failed
(gst-plugin-scanner:19): GLib-CRITICAL **: 10:07:49.012: g_once_init_leave: assertion 'result != 0' failed
(gst-plugin-scanner:19): GStreamer-CRITICAL **: 10:07:49.012: gst_element_register: assertion 'g_type_is_a (type, GST_TYPE_ELEMENT)' failed
(gst-plugin-scanner:19): GStreamer-WARNING **: 10:07:49.240: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnveglstreamsrc.so': /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnveglstreamsrc.so: file too short
(gst-plugin-scanner:19): GStreamer-WARNING **: 10:07:49.240: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvvideo4linux2.so': /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvvideo4linux2.so: file too short
(gst-plugin-scanner:19): GStreamer-WARNING **: 10:07:49.240: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvvidconv.so': /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvvidconv.so: file too short
(gst-plugin-scanner:19): GStreamer-WARNING **: 10:07:49.240: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvegltransform.so': /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvegltransform.so: file too short
(gst-plugin-scanner:19): GStreamer-WARNING **: 10:07:49.240: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnveglglessink.so': /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnveglglessink.so: file too short
(gst-plugin-scanner:19): GStreamer-WARNING **: 10:07:49.240: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvv4l2camerasrc.so': /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvv4l2camerasrc.so: file too short
(gst-plugin-scanner:19): GStreamer-WARNING **: 10:07:49.240: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvvideosink.so': /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvvideosink.so: file too short
(gst-plugin-scanner:19): GStreamer-WARNING **: 10:07:49.240: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvarguscamerasrc.so': /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvarguscamerasrc.so: file too short
(gst-plugin-scanner:19): GStreamer-WARNING **: 10:07:49.241: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvtee.so': /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvtee.so: file too short
(gst-plugin-scanner:19): GStreamer-WARNING **: 10:07:49.241: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvvideosinks.so': /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvvideosinks.so: file too short
(gst-plugin-scanner:19): GStreamer-WARNING **: 10:07:49.241: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvivafilter.so': /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvivafilter.so: file too short
(gst-plugin-scanner:19): GStreamer-WARNING **: 10:07:49.241: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvjpeg.so': /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvjpeg.so: file too short
(gst-plugin-scanner:19): GStreamer-WARNING **: 10:07:49.241: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvcompositor.so': /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvcompositor.so: file too short
(gst-plugin-scanner:19): GStreamer-WARNING **: 10:07:49.241: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvdrmvideosink.so': /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvdrmvideosink.so: file too short
gst-inspect-1.0 version 1.20.3
GStreamer 1.20.3
https://launchpad.net/distros/ubuntu/+source/gstreamer1.0
Sorry for the delay - yes, I believe so. Those gstreamer plugins normally get installed as drivers when JetPack is flashed, and mounted into the container. You should see them under /usr/lib/aarch64-linux-gnu/gstreamer-1.0/
If you continue to experience weirdness with this and need those plugins or encounter other issues with mounted drivers in the containers, I would recommend reflashing the device.