I am running Jetpack 5.1.1 / L4T 35.3.1on an AGX Orin DevKit. I used SDKManager to install the system on a 1TB NVMe SSD. And I am able to run Docker containers based on nvidia/cuda:11.4.1-devel-ubuntu20.04 just fine.
However I am unable to get Cuda 12.1 ‘passed’ to docker containers.
When running something like: nvidia-docker run -it -l "debug" --rm --gpus all --runtime=nvidia nvidia/cuda:12.1.1-devel-ubuntu22.04 bash
i end up with:
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: nvidia-container-runtime did not terminate successfully: exit status 1: time=“2023-06-16T14:33:19+02:00” level=error msg=“failed to create NVIDIA Container Runtime: failed to construct OCI spec modifier: requirements not met: unsatisfied condition: cuda>=12.1 (cuda=11.4) : unknown.”
However, my goal is to get a CUDA capable container based on ubuntu 22.04. Unfortunately the oldest cuda version for 22.04 that I could find in context of suitable docker base-images seems to be 11.7. which is not compatible with 11.4 provided via the nvidia-container-toolkit on the most recent Jetpack / L4T.
Is it somehow possible to obtain a Cuda 11.4 enabled 22.04 base image somewhere or force Docker / the nvidia-runtime to pass through Cuda 12.1 to containers on the jetson?
@Expert_Knob_Twiddler this container image isn’t built for JetPack (those are prefixed with l4t, like l4t-jetpack, l4t-cuda, ect)
On JetPack 5, CUDA/cuDNN/TensorRT/ect get installed into the container (unlike on JetPack 4 where they were mounted from the host device). My recommendation would be to start with l4t-base and install your desired version of CUDA Toolkit into it. You could also attempt mounting your /usr/local/cuda-xx.x directory into the container.
I’ve personally only used containers with GPU acceleration that are derived from the l4t-base container (which is based on Ubuntu 20.04 for JetPack 5 - Ubuntu 22.04 support is slated for JetPack 6). I imagine there are various hiccups that you could encounter attempting to rebase it against 22.04, including the fact that you are deviating from the tested/supported versions included in JetPack.
I recall on JetPack 4, some users dist-upgrading their l4t-base with apt with varying degrees of success. Good luck and let us know how it goes!
Thank you very much for the quick reply and clarifications. Also thank you for the pointer on the changes on how Cuda is handled in Jetpack 5. Is there a detailed technical documentation on the inner workings of this available somewhere?
When briefly testing, it appeared as if Cuda 11.4 seemed to work just fine, when using containers based on nvidia/cuda on top of L4T 35.x and Jetpack 5.1.1 using nvidia-docker2.
But coming back to your reply, this basically means there is currently no officially supported way of running cuda-enabled Ubuntu 22.04 (neither vanilla arm64 ubuntu or L4T-based) containers on the most recent Jetpack release 5.1.1, right?
Meanwhile, it appears a similar combination is running just fine on x86_64 Ubuntu 20.04 with Cuda 12.1 and a container based on Ubuntu 22.04. Obviously, it is using an entirely different nvidia driver family.
As an easy workaround is it possible to get a custom CUDA 11.4 binary either as .deb or as part of a docker base image for arm64 (that is Jetson compatible) based on ubuntu 22.04 somehow? Unfortunately I haven’t seen any containers from nvidia that support this combination. It would be great if the dockerfiles for x86_64 and arm64 coult be kept almost identical, so that it is straight forward to build a multiplatform docker image.
I also thought about going the dist-upgrade way from an l4t image based on 20.04 as well, but also for me, this appears to be a rather “unstable” solution.