Run docker images with cuda version different from the host cuda version

[kostastsig.94]

I am experiencing an error similar to this.

I have an AGX Orin module 32GB that has Jetpack 5.1.2 and CUDA 11.4 installed. I have upgraded the CUDA version to 12.2 following the guide here and when I try to run the docker image nvcr.io/nvidia/deepstream-l4t:7.0-samples-multiarch I am getting the error below:

docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: nvidia-container-runtime did not terminate successfully: exit status 1: time="2024-05-28T19:20:48+03:00" level=error msg="failed to create NVIDIA Container Runtime: failed to construct OCI spec modifier: requirements not met: unsatisfied condition: cuda>=12.2 (cuda=11.4)"

I have also set LD_LIBRARY_PATH=/usr/local/cuda-12.2/compat but the problem is not solved. Any ideas on how to solve this? Thanks in advance!

nvcc --version:
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Tue_Aug_15_22:08:11_PDT_2023
Cuda compilation tools, release 12.2, V12.2.140
Build cuda_12.2.r12.2/compiler.33191640_0

sudo nvidia-container-cli --load-kmods info
NVRM version: (null)
CUDA version: 11.4
Device Index: 0
Device Minor: 0
Model: Orin
Brand: (null)
GPU UUID: (null)
Bus Location: (null)
Architecture: 8.7

sudo jetson_release
Software part of jetson-stats 4.2.8 - (c) 2024, Raffaello Bonghi
Model: NVIDIA Jetson AGX Orin - AVerMedia D315 - Jetpack 5.1.2 [L4T 35.4.1]
NV Power Mode[0]: MAXN
Serial Number: [XXX Show with: jetson_release -s XXX]
Hardware:

• P-Number: p3701-0004
• Module: NVIDIA Jetson AGX Orin (32GB ram)
  Platform:
• Distribution: Ubuntu 20.04 focal
• Release: 5.10.120-tegra
  jtop:
• Version: 4.2.8
• Service: Active
  Libraries:
• CUDA: 12.2.140
• cuDNN: 8.6.0.166
• TensorRT: 8.5.2.2
• VPI: 2.3.9
• Vulkan: 1.3.204
• OpenCV: 4.5.4 - with CUDA: NO

[evandro.hauenstein]

I think this container is not suitable for your Jetpack version, you can try using Jetpack 6.0 (it includes CUDA 12.2.1)…

But this is probably a bug, it should work regardless of the installed CUDA version – for instance:

https://github.com/dusty-nv/jetson-containers/issues/258#issuecomment-1804648224

(dusty-nv managed to run CUDA 12.2 inside a docker container without having CUDA 12.2 installed)

Maybe the solution provided by dusty-nv can be applied to the deepstream container, but it will be required to build the container from sources

[kostastsig.94]

I am not sure that the solution you provided above applies to my case, as I only need to update the cuda version on the host PC (Jetson device) that I have already did and not inside the deepstream container. Also, Jetpack and CUDA version (11.4) are installed in the Jetson through the BSP provided by the manufacturer (AVerMedia) of the custom carrier board and the latest version of the BSP installs Jetpack 5.1.2.

[evandro.hauenstein]

According to your description you already have updated the CUDA version of your system, and both 11.4 and 12.2 are available for use.

You can double check by running the deviceQuery sample application, when setting LD_LIBRARY_PATH it will show 12.2 for both “CUDA Driver Version / Runtime Version” (fist line).

My point is that the container you are trying to use is for JetPack 6.0, see the prerequisites:

https://catalog.ngc.nvidia.com/orgs/nvidia/containers/deepstream-l4t

(it will not work with Jetpack 5)

The funny thing is that for some reason the container is checking the installed CUDA version, which IMHO makes no sense at all since it is possible to run CUDA 12.2 containers without installing this specific version (see also the link I have provided, just tested here and CUDA 12.2 worked, my Jetson system has CUDA 11.4 ONLY).