Docker engine fails when system started without a monitor

tamas.foldi · February 17, 2025, 4:05pm

I have a basic issue I cannot run docker containers with nvidia runtime unless I plug a monitor into the display port. It throws failed to stat CDI host device "/dev/fb0": no such file or directory: unknown as error.

When I start a docker container when the monitor is plugged in, all looks good

$ docker run --privileged --network host --ipc=host  -e DISPLAY -e NVIDIA_VISIBLE_DEVICES=all  -e NVIDIA_VISIBLE_DEVICES=nvidia.com/gpu=all   -ti --rm --runtime nvidia  nvidia/cuda:12.6.0-base-ubuntu22.04 nvidia-smi
Mon Feb 17 15:56:02 2025
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 540.4.0                Driver Version: 540.4.0      CUDA Version: 12.6     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Orin (nvgpu)                  N/A  | N/A              N/A |                  N/A |
| N/A   N/A  N/A               N/A /  N/A | Not Supported        |     N/A          N/A |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

However, when no monitor connected to my jetson device, all containers fail (so I cannot run Isaac Ros, etc.)

$ docker run --privileged --network host --ipc=host  -e DISPLAY -e NVIDIA_VISIBLE_DEVICES=all  -e NVIDIA_VISIBLE_DEVICES=nvidia.com/gpu=all   -ti --rm --runtime nvidia  nvidia/cuda:12.6.0-base-ubuntu22.04
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: could not apply required modification to OCI specification: error modifying OCI spec: failed to inject CDI devices: failed to inject devices: failed to stat CDI host device "/dev/fb0": no such file or directory: unknown.

I am sure I am the dumb here, but how can I configure the device that it will run nvidia containers regardless if a monitor is plugged in or not?

AastaLLL · February 18, 2025, 7:13am

Hi,

Removing the -e NVIDIA_VISIBLE_DEVICES=nvidia.com/gpu=all can work correctly.

$ docker run --privileged --network host --ipc=host -e DISPLAY -e NVIDIA_VISIBLE_DEVICES=all -ti --rm --runtime nvidia  nvidia/cuda:12.6.0-base-ubuntu22.04 nvidia-smi
Tue Feb 18 07:11:07 2025       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 540.4.0                Driver Version: 540.4.0      CUDA Version: 12.6     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Orin (nvgpu)                  N/A  | N/A              N/A |                  N/A |
| N/A   N/A  N/A               N/A /  N/A | Not Supported        |     N/A          N/A |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

On Jetson, you don’t need to set that variable as the Orin series only has one GPU.
So it should not impact the functionality.

Thanks

tamas.foldi · February 18, 2025, 2:48pm

Thank you, yes, it works in that case.

My main issue is that by default this flag is set by run_dev.sh from isaac common:

github.com/NVIDIA-ISAAC-ROS/isaac_ros_common

scripts/run_dev.sh

04d1036a1


      
          DOCKER_ARGS+=("-e HOST_USER_UID=`id -u`")
          DOCKER_ARGS+=("-e HOST_USER_GID=`id -g`")
          
          # Forward SSH Agent to container if the ssh agent is active.
          if [[ -n $SSH_AUTH_SOCK ]]; then
              DOCKER_ARGS+=("-v $SSH_AUTH_SOCK:/ssh-agent")
              DOCKER_ARGS+=("-e SSH_AUTH_SOCK=/ssh-agent")
          fi
          
          if [[ $PLATFORM == "aarch64" ]]; then
              DOCKER_ARGS+=("-e NVIDIA_VISIBLE_DEVICES=nvidia.com/gpu=all,nvidia.com/pva=all")
              DOCKER_ARGS+=("-v /usr/bin/tegrastats:/usr/bin/tegrastats")
              DOCKER_ARGS+=("-v /tmp/:/tmp/")
              DOCKER_ARGS+=("-v /usr/lib/aarch64-linux-gnu/tegra:/usr/lib/aarch64-linux-gnu/tegra")
              DOCKER_ARGS+=("-v /usr/src/jetson_multimedia_api:/usr/src/jetson_multimedia_api")
              DOCKER_ARGS+=("--pid=host")
              DOCKER_ARGS+=("-v /usr/share/vpi3:/usr/share/vpi3")
              DOCKER_ARGS+=("-v /dev/input:/dev/input")
          
              # If jtop present, give the container access
              if [[ $(getent group jtop) ]]; then

I guess I can remove that line. Or shall I check what the system is and remove this line based on the host type?

jolin · February 20, 2025, 8:27am

Hi @tamas.foldi,

Could you please help to run the below command outside the container and try again?

sudo nvidia-ctk cdi generate --mode=csv --output=/etc/cdi/nvidia.yaml

Refer to: Jetson Setup for VPI — isaac_ros_docs documentation

Thanks.

tamas.foldi · February 21, 2025, 7:24am

tamas.foldi:

docker run --privileged --network host --ipc=host  -e DISPLAY -e NVIDIA_VISIBLE_DEVICES=all  -e NVIDIA_VISIBLE_DEVICES=nvidia.com/gpu=all   -ti --rm --runtime nvidia  nvidia/cuda:12.6.0-base-ubuntu22.04

this helped, thanks!

system · March 7, 2025, 7:24am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
ISAAC ROS Docker failed to inject CDI devices: unresolvable CDI devices nvidia.com/pva=all: unknown Isaac ROS docker	3	76	January 26, 2025
`nvidia-container-cli` driver error when trying to run Nvidia docker on Jetson Nano Jetson Nano cuda , containers	6	7366	October 18, 2021
Cannot run docker with --runtime nvidia Jetson Xavier NX docker , containers	8	7514	December 22, 2021
ISAAC ROS Docker failed to inject CDI devices: unresolvable CDI devices nvidia.com/gpu=all: unknown Isaac ROS docker , isaac-ros-common	6	84	February 7, 2025
Jetson TX2 Unable to access or use gpu from docker Jetson TX2 docker , containers	3	1566	October 18, 2021
Docker issue on permission? Jetson AGX Orin docker	5	163	November 22, 2024
Unable to run nvidia docker Jetson Xavier NX docker	4	3611	December 8, 2021
Use Jetson AGX Orin’s GPU from Rootless Docker Jetson AGX Orin docker	11	854	June 6, 2024
Rootless Docker; ERROR: No supported GPU(s) detected to run this container Docker and NVIDIA Docker docker	2	7683	April 8, 2022
Isaac ROS Setup: Docker launch failed Isaac ROS jetson	7	422	December 16, 2024

Docker engine fails when system started without a monitor

Related topics