We are building a PREEMPT_RT kernel on Jetson Orin Nano using the official build script and have a question about CUDA support on the resulting RT kernel.
Environment:
Hardware: Jetson Orin Nano
JetPack: 6.2.2
RT kernel built with: ./nvbuild.sh -r -m (resulting in kernel 5.15.185-rt-tegra)
CUDA Runtime: 12.6
Observed behavior:
After booting the RT kernel and running the CUDA deviceQuery sample, it fails with error 801:
$ ./deviceQuery
./deviceQuery Starting…
CUDA Device Query (Runtime API) version (CUDART static linking)
cudaGetDeviceCount returned 801
→ unknown error
Result = FAIL
The same binary runs successfully on the standard (non-RT) kernel included with JetPack 6.2.2.
Question:
Is CUDA expected to work on the PREEMPT_RT kernel when built using nvbuild.sh -r? Is this a known limitation, or is there official guidance on running CUDA workloads alongside a
real-time kernel on Jetson Orin?
I first tried running nvidia-smi, and the output was:
$ sudo nvidia-smi
Unable to determine the device handle for GPU0002:00:00.0: Unknown Error
More, could you share the detailed steps you used for setup so we can check if there are any issues?
I created a bash script to reproduce the issue and ran it on Ubuntu 22.04 running under WSL2 for cross-compilation.
#!/bin/bash
set -euxo pipefail
## define environment variables
SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &> /dev/null && pwd)
export CROSS_COMPILE=/usr/bin/aarch64-linux-gnu-
export INSTALL_MOD_PATH="${SCRIPT_DIR}/Linux_for_Tegra/rootfs"
export IGNORE_PREEMPT_RT_PRESENCE=1
export KERNEL_HEADERS="${SCRIPT_DIR}/Linux_for_Tegra/source/kernel/kernel-jammy-src"
## download necessary tarballs
wget https://developer.nvidia.com/downloads/embedded/l4t/r36_release_v5.0/sources/public_sources.tbz2
wget https://developer.nvidia.com/downloads/embedded/l4t/r36_release_v5.0/release/Jetson_Linux_r36.5.0_aarch64.tbz2
wget https://developer.nvidia.com/downloads/embedded/l4t/r36_release_v5.0/release/Tegra_Linux_Sample-Root-Filesystem_r36.5.0_aarch64.tbz2
## expand tarballs
tar jxvf Jetson_Linux_r36.5.0_aarch64.tbz2
tar jxvf public_sources.tbz2
## expand rootfs
cp Tegra_Linux_Sample-Root-Filesystem_r36.5.0_aarch64.tbz2 Linux_for_Tegra/rootfs/
cd Linux_for_Tegra/rootfs
tar jxvf Tegra_Linux_Sample-Root-Filesystem_r36.5.0_aarch64.tbz2
## move to source directory
cd ../source
tar jxvf kernel_src.tbz2
tar jxvf kernel_oot_modules_src.tbz2
tar jxvf nvidia_kernel_display_driver_source.tbz2
## build and install to rootfs
./nvbuild.sh -r
./nvbuild.sh -r -i
## scp kernel image and modules to jetson
cd ../rootfs
## edit this variable
remote_work_dir_path=
cd boot
scp Image jetson:${remote_work_dir_path}/Image-rt
cd ../lib/modules
sudo tar jcvf 5.15.185-rt-tegra.tbz2 5.15.185-rt-tegra
scp 5.15.185-rt-tegra.tbz2 jetson:${remote_work_dir_path}
Next, I installed the kernel image and the modules on the Jetson Orin Nano using the following script. This script is located in the same directory as the kernel image and modules.
#!/bin/bash
set -euxo pipefail
sudo cp Image-rt /boot/
tar jxvf 5.15.185-rt-tegra.tbz2
sudo cp -r 5.15.185-rt-tegra /lib/modules/
sudo depmod -a
sudo update-initramfs -c -k 5.15.185-rt-tegra
sudo cp /boot/initrd.img-5.15.185-rt-tegra /boot/initrd-rt
My extlinux.conf is as follows. Note that I had already enabled the I2S function using jetson-io.py, so the FDT and OVERLAYS entries were added.