ISAAC ROS Docker failed to inject CDI devices: unresolvable CDI devices nvidia.com/gpu=all: unknown

Robotics & Edge Computing Robotics - Isaac Isaac ROS
,
1

Hey everyone. I just did a fresh install of Jetson Linux and JetPack 6.2 through the SDK Manager on my Jetson AGX Orin 32GB Developer Kit. I followed the Development Environment setup in the ISAAC ROS documentation. And configured my ISAAC ROS Dev environment for running ISAAC ROS through Docker with the isaac_ros_common code pulled from GitHub. When I try to run the container, I am getting the following error output:

haf@jetson-orin:~/isaac_ros-dev/src/isaac_ros_common/scripts$ ./run_dev.sh 
Launching Isaac ROS Dev container with image key aarch64.ros2_humble: /home/haf/isaac_ros-dev/
Building aarch64.ros2_humble base as image: isaac_ros_dev-aarch64
Building layered image for key aarch64.ros2_humble as isaac_ros_dev-aarch64
Using configured docker search paths: /home/haf/isaac_ros-dev/src/isaac_ros_common/scripts/../docker
Checking if base image nvcr.io/nvidia/isaac/ros:aarch64-ros2_humble_e2b088db4f4af4f7f4e886aee5c0d9fa exists on remote registry
Found pre-built base image: nvcr.io/nvidia/isaac/ros:aarch64-ros2_humble_e2b088db4f4af4f7f4e886aee5c0d9fa
aarch64-ros2_humble_e2b088db4f4af4f7f4e886aee5c0d9fa: Pulling from nvidia/isaac/ros
Digest: sha256:1dbb3f36952fdbd6122e25e9ca7aef1ab215cb36a68c83f3834f2116b90d9790
Status: Image is up to date for nvcr.io/nvidia/isaac/ros:aarch64-ros2_humble_e2b088db4f4af4f7f4e886aee5c0d9fa
nvcr.io/nvidia/isaac/ros:aarch64-ros2_humble_e2b088db4f4af4f7f4e886aee5c0d9fa
Finished pulling pre-built base image: nvcr.io/nvidia/isaac/ros:aarch64-ros2_humble_e2b088db4f4af4f7f4e886aee5c0d9fa
Nothing to build, retagged nvcr.io/nvidia/isaac/ros:aarch64-ros2_humble_e2b088db4f4af4f7f4e886aee5c0d9fa as isaac_ros_dev-aarch64
Using additional Docker run arguments from /home/haf/.isaac_ros_dev-dockerargs
Running isaac_ros_dev-aarch64-container
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: could not apply required modification to OCI specification: error modifying OCI spec: failed to inject CDI devices: unresolvable CDI devices nvidia.com/gpu=all: unknown.
~/isaac_ros-dev/src/isaac_ros_common/scripts

We were previously running JetPack 6.1 on this same system with a ISAAC ROS Dev environment setup and the Docker container running. We ran this setup without issues for weeks. The error described above appeared for the first time on Friday before we tried reinstalling the OS and JetPack. Assuming it had something to do with the JetPack 6.2 release or perhaps a configuration problem we made, we tried the reinstall described above. And even running everything stock from Nvidia without any modifications, we are sill seeing this issue. Has anybody else experienced this issue or have any ideas on what is causing this problem? Any insight would be appreciated.

4

Hi @riley.mathiasen ,

Thanks for your post and welcome to the Isaac ROS forum.

Please try below command which should fix the problem:

sudo nvidia-ctk cdi generate --mode=csv --output=/etc/cdi/nvidia.yaml

Thanks.

5

Thank you very much! This solved my issue.

6

Hi, @jolin ,
Thanks for the solution, I tried your command, but there is still error:
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: could not apply required modification to OCI specification: error modifying OCI spec: failed to inject CDI devices: unresolvable CDI devices nvidia.com/pva=all: unknown.

And the output of sudo nvidia-ctk cdi list is:

INFO[0000] Found 2 CDI devices
nvidia.com/gpu=0
nvidia.com/gpu=all

7

Hi @absorbraver ,

Please help to confirm the command is executed outside the docker container.
If still have problem, please file new topic and provide more detail, thank you.

Regards.

8

OK, i opened a new topic here: ISAAC ROS Docker failed to inject CDI devices: unresolvable CDI devices nvidia.com/pva=all: unknown

9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.