Trouble running docker on Xavier

Autonomous Machines Jetson & Embedded Systems Jetson AGX Xavier
,
1

Hello,

System:

  • Jetson Linux 35.4.1, on Xavier AGX industrial

we are trying to run nvidia-docker (docker with nvidia runtime) on a Jetson Xavier, but are failing with this error message:

“docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: requirement error: unsatisfied condition: cuda>=11.5, please update your driver to a newer version, or use an earlier cuda container: unknown.”

The Docker image is based on “nvidia/cuda:11.5.2-cudnn8-devel-ubuntu20.04”. The image was built on a x86 host using qemu.

Since CUDA 11.4 is default on the BSP, we upgraded CUDA on the Jetson by following this guide:

And the app note here: CUDA for Tegra

We set LD_LIBRARY_PATH=/usr/local/cuda-12.2/compat and did a test and compiled the deviceQuery example and this gives (only showing the first lines of the output):

"root@linux:/home/dev/tmp2/deviceQuery# ./deviceQuery

./deviceQuery Starting…

CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: “Xavier”

CUDA Driver Version / Runtime Version 12.2 / 12.2

CUDA Capability Major/Minor version number: 7.2

…"

So it looks like the CUDA-driver has been upgraded, but we are still not able to run Docker. Any hints on why this happen? We are using the same LD_LIBRARY_PATH

as when starting deviceQuery, but not sure if Docker is able to pick up the cuda version from there.

Simplifying CUDA Upgrades for NVIDIA Jetson Users | NVIDIA Technical Blog

With CUDA Toolkit 11.8 and NVIDIA JetPack 5.0, you can upgrade to the latest CUDA release without updating NVIDIA JetPack or Jetson Linux BSP software.

All help is deeply appreciated.

Thanks!

4

Hi,

Could you test the l4t based container?
The environment between qemu and Jetson is still different, especially on the GPU driver.

Thanks.

5

Hi, and thanks for replying to us so quickly.

We tried to use the container you suggested, and we are able to start that container, however it is running CUDA 11.4, and we need to run a newer version.

So we are wondering how we should proceed in order to get a newer CUDA version inside the container? We have tried to follow the guide:

Inside the conatiner, but we failed to get it working.
Is it supposed to work? and if so do you know a guide / tutorial on how to do this?

Is there any other way to get the desired cuda version inside the conatiner?

A little extra question: is there any dependency between CUDA version in conatiner and host?

Thank you :)

6

Hi,

You can follow the instructions below on the top of the l4t container:

The latest CUDA toolkit has the compatibility feature which means it can run on an older GPU driver.

Thanks

8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.