Jetson powering method causes errors with camera detection

Robotics & Edge Computing Robotics - Isaac Isaac ROS
, , ,
1

Hello,

I am using a Jetson Orin Dev Kit and a ZED camera, and sometimes the camera is not detected as a CUDA capable device and can’t be opened. The following command fails with an error :

> /usr/local/cuda/bin/__nvcc_device_query
/usr/local/cuda/bin/__nvcc_device_query failed to call cudaLoader::cuInit(0) with error 0x64 (CUDA_ERROR_NO_DEVICE)

This depends on the way I turn the Jetson board on :

  • If I perform a soft reboot with sudo reboot now I have no issue
  • If I perform a soft shutdown and restart the board with the power button, I get the error

My environment is the following :

  • Nvidia Jetson Orin with Jetpack 7.2
  • Isaac ROS 4.6.0.0 installed in docker
  • Added the default ZED layer to the docker image in order to install ZED SDK v5.2 and ZED ROS2 wrapper v5.2
  • ZED Driver v1.4.3
  • ZED X camera plugged to the Jetson Devkit using a ZED Link Mono capture card

I noticed a few things about this issue.

First, the camera is always correctly detected outside of the docker container, according to the dmesg logs :

[   17.701310] sl_max9296 10-0048: sl_max9296_get_cam: zedx camera identified via eeprom
[   17.701564] sl_max9296 10-0048: sl_max9296_probe: zedx camera connected to this port
[   19.150443] sl_max9295 10-0062: probe_serializer: Serializer for zedx detect success
[   19.171783] zedx 10-0020: Driver Version : v1.4.3
[   19.172088] zedx 10-0020: tegracam sensor driver:video2_v2.0.6
[   19.405037] tegra-camrtc-capture-vi tegra-capture-vi: subdev zedx 10-0020 bound
[   19.412820] zedx 10-0020: ZED-X sensor initialisation done
[   19.529241] zedx 10-0020: zedx_probe: Serial Number : 48331234
[   19.529300] zedx 10-0020: zedx_probe: success
[   19.530207] zedx 10-0028: Driver Version : v1.4.3
[   19.530359] zedx 10-0028: tegracam sensor driver:video0_v2.0.6
[   19.772642] tegra-camrtc-capture-vi tegra-capture-vi: subdev zedx 10-0028 bound
[   19.773675] zedx 10-0028: ZED-X sensor initialisation done
[   19.897462] zedx 10-0028: zedx_probe: Serial Number : 48331234
[   19.897569] zedx 10-0028: zedx_probe: success

Secondly, when the issue occurs I can always see that the ownership of the /dev/nvhost* files are set to root:root instead of root:video inside the docker container.
For example, with the file /dev/nvhost-gpu I have the following permissions with a working system :

crw-rw---- 1 root video 493,   1 Sep 23 15:56 /dev/nvhost-gpu

But I get this when it fail to open the camera :

crw------- 1 root root 493,   1 Sep 23 15:56 /dev/nvhost-gpu

Lastly, I noticed the issue never happens when I plug a monitor to my dev kit. However as soon as I plug it out and restart the board using the power button, it fails again.

It’s also interesting to note that this error does not occurs on every power button boot, it feels quite random

This makes it quite complicated to debug, and I was hoping someone could help me with this here ?

Thank you for your time,
BR

2

Hello @_paulm,

Thank you for providing the working and failing container permissions.

Because the ZED X probe succeeds but CUDA initialization fails, we would like to determine whether the incorrect device permissions originate on the Jetson host or only inside the container.

After reproducing a failing headless cold boot, please run the following before rebooting, reloading udev rules, or restarting the container and share the complete output:

ls -l /dev/nvhost-gpu /dev/nvhost-ctrl-gpu /dev/nvmap
grep -nE 'nvhost|nvmap' /etc/udev/rules.d/99-tegra-devices.rules

Please also run the CUDA query once as the container’s normal user and once as root.

This will allow us to distinguish between:

  • A Jetson Linux udev/device-initialization issue.
  • A container device-mapping or startup-order issue.
  • A container user/group permission issue.
3

Hello @vchuang, thank you for your answer !

Here is the output you asked for (ran inside the container) :

> ls -l /dev/nvhost-gpu /dev/nvhost-ctrl-gpu /dev/nvmap

crw------- 1 root root  493,   2 Jan  1  1970 /dev/nvhost-ctrl-gpu
crw------- 1 root root  493,   1 Jan  1  1970 /dev/nvhost-gpu
cr--r----- 1 root video  10, 261 Jan  1  1970 /dev/nvmap

The same command ran outside of the Docker container returns this :

> ls -l /dev/nvhost-gpu /dev/nvhost-ctrl-gpu /dev/nvmap

crw-rw---- 1 root video 493,   2 Jan  1  1970 /dev/nvhost-ctrl-gpu
crw-rw---- 1 root video 493,   1 Jan  1  1970 /dev/nvhost-gpu
cr--r----- 1 root video  10, 261 Jan  1  1970 /dev/nvmap

The file /etc/udev/rules.d/99-tegra-devices.rules exists on my Jetson but it does not exist inside my Docker container, so I could only run this command outside of Docker :

> grep -nE 'nvhost|nvmap' /etc/udev/rules.d/99-tegra-devices.rules

31:KERNEL=="knvmap" OWNER="root" GROUP="root" MODE="0660"
45:KERNEL=="nvmap" OWNER="root" GROUP="video" MODE="0440"
50:KERNEL=="nvhost*" OWNER="root" GROUP="video" MODE="0660"
51:KERNEL=="nvhost-ctxsw-gpu" OWNER="root" GROUP="debug" MODE="0660"
52:KERNEL=="nvhost-dbg-gpu" OWNER="root" GROUP="debug" MODE="0660"
53:KERNEL=="nvhost-prof-ctx-gpu" OWNER="root" GROUP="debug" MODE="0660"
54:KERNEL=="nvhost-prof-dev-gpu" OWNER="root" GROUP="debug" MODE="0660"
55:KERNEL=="nvhost-prof-gpu" OWNER="root" GROUP="debug" MODE="0660"
56:KERNEL=="nvhost-sched-gpu" OWNER="root" GROUP="root" MODE="0660"

Finally, running the CUDA query as the normal user returns this :

> /usr/local/cuda/bin/__nvcc_device_query

/usr/local/cuda/bin/__nvcc_device_query failed to call cudaLoader::cuInit(0) with error 0x64 (CUDA_ERROR_NO_DEVICE)

Running the same query as root returns a valid integer value :

> /usr/local/cuda/bin/__nvcc_device_query

87

Please let me know if you need any additional information. Thank you for your support !

BR

4

If you aren’t already you may want to add

docker run -it --rm --privileged --runtime=nvidia --gpus=all \

Here’s a script I use to run zed camera and ros in docker

cat isaac-zed-docker.sh 
#!/usr/bin/env bash
# Start the Isaac ROS + ZED container, or open another shell in it if it is
# already running. Extra arguments replace the default command (bash) and run
# with the ROS environment loaded, e.g.:
#   isaac-zed-docker.sh /workspaces/isaac_ros-dev/scripts/zed-up.sh   (Orin)
#   isaac-zed-docker.sh rviz2                                         (Thor)
set -euo pipefail

IMAGE="isaac_ros_dev-zed:latest"
NAME="isaac-zed"
WS="${ISAAC_ROS_WS:-$HOME/workspaces/isaac_ros-dev}"
ZED_SETTINGS="$HOME/.zed/settings"
ZED_RESOURCES="$HOME/.zed/resources"
RVIZ_CONFIG="$HOME/.rviz2"

# Run commands through an interactive bash so /etc/bash.bashrc sources ROS and
# the wrapper workspace, as in a normal shell in the container.
if [ "$#" -eq 0 ]; then
  set -- bash
else
  set -- bash -ic 'exec "$@"' bash "$@"
fi

# A second terminal joins the running container instead of failing on the name.
if [ -n "$(docker ps -q -f "name=^${NAME}$")" ]; then
  exec docker exec -it "$NAME" "$@"
fi

# X11: pass the X authority file into the container so its root user can open windows.
X11_ARGS=()
if [ -n "${DISPLAY:-}" ]; then
  XAUTH_FILE="${XAUTHORITY:-$HOME/.Xauthority}"
  if [ -f "$XAUTH_FILE" ]; then
    X11_ARGS+=(-v "$XAUTH_FILE":/root/.Xauthority:ro -e XAUTHORITY=/root/.Xauthority)
  fi
  case "$DISPLAY" in
    :*) xhost +si:localuser:root > /dev/null 2>&1 || true ;;
  esac
else
  echo "DISPLAY is not set: GUI tools (rviz2, ZED_Explorer) will not open." >&2
  echo "Connect with 'ssh -X' to forward windows to your own machine." >&2
fi

# The ZED SDK downloads the camera calibration here on first use; keeping it on
# the host means it survives the container.
mkdir -p "$ZED_SETTINGS"

# rviz2 saves its layout in /root/.rviz2; keep it so it survives the container.
mkdir -p "$RVIZ_CONFIG"

# Mount the Isaac ROS workspace only on machines that have one.
WS_ARGS=()
if [ -d "$WS" ]; then
  WS_ARGS=(-v "$WS":/workspaces/isaac_ros-dev -w /workspaces/isaac_ros-dev)
fi

# Neural depth models: the SDK optimizes them for this GPU on first use, which
# takes several minutes, so keep the results on the host too. Seed the folder
# with the models shipped in the image the first time.
if [ ! -d "$ZED_RESOURCES" ] || [ -z "$(ls -A "$ZED_RESOURCES")" ]; then
  mkdir -p "$ZED_RESOURCES"
  docker run --rm --runtime=runc --entrypoint bash -v "$ZED_RESOURCES":/out "$IMAGE" \
    -c 'cp -r /usr/local/zed/resources/* /out/ 2>/dev/null || true'
fi

exec docker run -it --rm --name "$NAME" \
  --runtime=nvidia --gpus=all --privileged \
  --network host --ipc=host \
  --ulimit memlock=-1 --ulimit stack=67108864 \
  -e DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix "${X11_ARGS[@]}" \
  -e ROS_DOMAIN_ID -e ROS_AUTOMATIC_DISCOVERY_RANGE -e ROS_STATIC_PEERS -e RMW_IMPLEMENTATION \
  -v /dev:/dev \
  -v "$ZED_SETTINGS":/usr/local/zed/settings \
  -v "$ZED_RESOURCES":/usr/local/zed/resources \
  -v "$RVIZ_CONFIG":/root/.rviz2 \
  "${WS_ARGS[@]}" \
  "$IMAGE" "$@"
5

Hello @_paulm,

Thank you for the details. The host GPU nodes have the expected root:video 0660 permissions, while /dev/nvhost-gpu and /dev/nvhost-ctrl-gpu inside the container are root:root 0600. CUDA succeeding as root supports a container device-permission issue.

Since the failure occurs after shutdown and power-button startup, but not after sudo reboot, please try the following comparison to check whether container startup timing contributes:

  1. Reproduce the failing headless boot, then run this on the host:

    stat -Lc '%n mode=%a uid=%u gid=%g' /dev/nvhost-gpu /dev/nvhost-ctrl-gpu /dev/nvmap

  2. After confirming the host GPU nodes have mode 660 and belong to the video group, create a fresh container using the same image and launch options, without rebooting the Jetson. Please ensure this creates a new container rather than reconnecting to the existing one.

  3. Inside the fresh container, run these commands as the normal user:
    id
    stat -Lc '%n mode=%a uid=%u gid=%g' /dev/nvhost-gpu /dev/nvhost-ctrl-gpu /dev/nvmap /usr/local/cuda/bin/__nvcc_device_query

If the fresh container has the correct permissions and CUDA works, that would support a startup-timing or stale device-metadata explanation. If it still shows 600 root:root, we should inspect the container launch/runtime configuration.

Please share the outputs, your launch command, and whether the original container starts automatically at boot. The launch script shared above may also be useful for comparison, but please test CUDA as the same normal user.

Also, similar non-root CUDA permission failures have reportedly disappeared after reflashing before. If the issue persists and you can back up your data and reinstall your environment, a clean JetPack reflash remains an optional recovery path, although it is not a confirmed fix for this case.

6

Hello @vchuang, @whitesscott

Thank you for your replies.

Here are the results of the commands :

> stat -Lc '%n mode=%a uid=%u gid=%g' /dev/nvhost-gpu /dev/nvhost-ctrl-gpu /dev/nvmap
/dev/nvhost-gpu mode=660 uid=0 gid=44
/dev/nvhost-ctrl-gpu mode=660 uid=0 gid=44
/dev/nvmap mode=440 uid=0 gid=44

I created another container with isaac-ros activate --config docker.run.container_name=debug and ran the commands inside it :

> id
uid=1000(admin) gid=1000(admin) groups=1000(admin),4(adm),20(dialout),24(cdrom),25(floppy),27(sudo),29(audio),30(dip),44(video),46(plugdev),108(hwgrp108),984(hwgrp984),993(render)

> stat -Lc '%n mode=%a uid=%u gid=%g' /dev/nvhost-gpu /dev/nvhost-ctrl-gpu /dev/nvmap /usr/local/cuda/bin/__nvcc_device_query
/dev/nvhost-gpu mode=600 uid=0 gid=0
/dev/nvhost-ctrl-gpu mode=600 uid=0 gid=0
/dev/nvmap mode=440 uid=0 gid=44
/usr/local/cuda/bin/__nvcc_device_query mode=755 uid=0 gid=0

I am using isaac-ros activate to run my container. It does not start automatically at boot, I am running it manually for now.
Here is the full launch command generated by isaac-ros-cli :

docker run --name=isaac_ros_dev_container --hostname=jetson-orin-0002 --pid host \
--entrypoint /usr/local/bin/scripts/workspace-entrypoint.sh \
--volume /tmp/.X11-unix:/tmp/.X11-unix --volume /usr/bin/tegrastats:/usr/bin/tegrastats --volume /home/jetson/.profile:/home/admin/.profile:ro --volume /home/jetson/.config:/home/admin/.config --volume /dev/bus/usb:/dev/bus/usb --volume /home/jetson/.ssh:/home/admin/.ssh:ro --volume /usr/src/jetson_sipl_api:/usr/src/jetson_sipl_api --volume /usr/share/vpi3:/usr/share/vpi3 --volume /usr/lib/aarch64-linux-gnu/tegra:/usr/lib/aarch64-linux-gnu/tegra --volume /tmp:/tmp --volume /etc/localtime:/etc/localtime:ro --volume /home/jetson/.bashrc:/home/admin/.bashrc:ro --volume /sys/kernel/debug:/sys/kernel/debug:ro --volume /home/jetson/workspaces/isaac_ros-dev:/workspaces/isaac_ros-dev --volume /usr/src/jetson_multimedia_api:/usr/src/jetson_multimedia_api --volume /dev/input:/dev/input --volume /home/jetson/.Xauthority:/home/admin/.Xauthority --volume /home/jetson/.aws:/home/admin/.aws:ro --volume /home/jetson/.bash_history:/home/admin/.bash_history --volume /home/jetson/.cache:/home/admin/.cache --volume /home/jetson/.gitconfig:/home/admin/.gitconfig \
--env=TERM=xterm-256color --env=COLORTERM=truecolor --env=FORCE_COLOR=true --env=ISAAC_ROS_PLATFORM=arm64-jetpack --env=DISPLAY --env=NVIDIA_VISIBLE_DEVICES=all --env=NVIDIA_DRIVER_CAPABILITIES=all --env=HOST_USER_UID=1000 --env=ROS_DOMAIN_ID --env=USER=jetson --env=ISAAC_ROS_WS=/workspaces/isaac_ros-dev --env=HOST_USER_GID=1000 \
--network=host --privileged --workdir=/workspaces/isaac_ros-dev --runtime=nvidia -t --rm nvcr.io/nvidia/isaac/ros:isaac_ros-zed_dfd1d0b0f71cede03fdec9e248d6b0d4-arm64-jetpack /bin/bash

This is similar to the command from the script shared by @whitesscott, but it’s missing --gpus=all, is it a problem ?
I will try the script and see how it goes.

Thank you for your support,
BR

7

Your instinct is right --gpus all could be the fix. JetPack 7.2 uses CDI mode in the NVIDIA Container Toolkit, which requires an explicit --gpus all to properly inject GPU devices into the container. Without it, /dev/nvhost-gpu appears as 600 root:root instead of 660 root:video, and CUDA can’t initialize.

Looking at the launch command you shared, it’s using --runtime=nvidia without --gpus all. This indicates your isaac-ros-cli is older than 4.3, from 4.3 onwards, run_dev.py uses get_gpu_docker_args() which auto-selects --gpus all for Jetson Orin.

Please update the isaac-ros cli to the latest version and restart the container:
sudo apt update && sudo apt install --only-upgrade isaac-ros-cli

You can also confirm the installed version with:
dpkg -l isaac-ros-cli

8

Ok, but I don’t really understand how versions are numbered because dpkg -l isaac-ros-cli returns 2.5.0-1.20260818191437 which does not match any released version (according to the Github repo).

I specifically installed isaac-ros 4.6 with the following commands from the documentation :

k="/usr/share/keyrings/nvidia-isaac-ros.gpg"
curl -fsSL https://isaac.download.nvidia.com/isaac-ros/repos.key | sudo gpg --dearmor \
    | sudo tee -a $k > /dev/null
f="/etc/apt/sources.list.d/nvidia-isaac-ros.list"
sudo touch $f
s="deb [signed-by=$k] https://isaac.download.nvidia.com/isaac-ros/release-4.6 noble-jetpack main"
grep -qxF "$s" $f || echo "$s" | sudo tee -a $f

sudo apt-get update

And then ran

sudo apt update && sudo apt install isaac-ros-cli

When running the upgrade command, I’m getting a message saying my version is already the newest one available :

isaac-ros-cli is already the newest version (2.5.0-1.20260818191437).

I guess the apt package numbering is different from the Github releases numbering ? How can I force apt to install a specific isaac-ros-cli version ?

Thank you for your support,
BR

9

If everything else is isaac-ros 4.6 then 2.5.0-1 is correct version:

github.com/NVIDIA-ISAAC-ROS/isaac-ros-cli/blob/main/debian/changelog

isaac-ros-cli (2.5.0-1) noble; urgency=low
  * Add configurable Docker image push defaults and `isaac-ros activate --push` and `--no-push` overrides.
  * Improve container access to I2C, GPIO, DRM render, and GMSL frame synchronization devices.
  * Update CUDA, TensorRT, Jetson, and DGX Spark package selection for JetPack 7.2 and Dockerless workflows.
  * Update Isaac Teleop and pyvers pip shim constraints, rosdep mappings, 

  * Docker image definitions for Isaac ROS 4.6.
10

We tested Isaac ROS 4.6 on a Jetson Orin with JetPack 7.2, including the default ZED SDK/wrapper 5.2.0 layer. We did not observe the GPU permission issue you described: the GPU nodes had the expected 0660 root:video permissions, and CUDA worked as the normal container user.

We also compared your Docker launch command with ours and noticed differences. Your command includes --runtime=nvidia without --gpus all, while our CLI-generated command includes --gpus all and --ipc=host.

Could you confirm how you obtained the Docker command you shared and provide the complete output of the following command from a fresh container launch? isaac-ros activate --verbose

This will help us compare the launch configuration and investigate the permission difference.