Hi all,
I am trying to use the kernel customisation to bring up a custom Thor carrier board. The resultant build flashes and boots succcesfully (on the Developer Kit and on the custom carrier), however running nvidia-smi results in No devices found. What steps could I take to troubleshoot why the gpu driver doesn’t seem to be built/loaded? The only thing I can see relating to the gpu on dmesg is:
[ 16.106675] [drm] [nvidia-drm] [GPU ID 0x00000100] Loading driver
[ 16.174334] NVRM: GPU 0000:01:00.0: RmInitAdapter failed! (0x61:0x56:1893)
[ 16.174499] NVRM: GPU 0000:01:00.0: rm_init_adapter failed, device minor number 0
[ 16.174639] [drm:nv_drm_dev_load [nvidia_drm]] ERROR [nvidia-drm] [GPU ID 0x00000100] Failed to allocate NvKmsKapiDevice
[ 16.180965] [drm:nv_drm_register_drm_device [nvidia_drm]] ERROR [nvidia-drm] [GPU ID 0x00000100] Failed to load device
[ 16.191831] [drm] [nvidia-drm] [GPU ID 0x00020000] Loading driver
[ 16.441315] nvidia-modeset: WARNING: GPU:0: The head configuration (0xff) is inconsistent with the number of heads (4)
[ 18.190010] [drm] [nvidia-drm] [GPU ID 0x00020000] Invalid framebuffer console info
[ 19.195074] NVRM: GPU 0000:01:00.0: RmInitAdapter failed! (0x61:0x56:1893)
[ 19.195367] NVRM: GPU 0000:01:00.0: rm_init_adapter failed, device minor number 0
[ 29.390098] NVRM: GPU 0000:01:00.0: RmInitAdapter failed! (0x61:0x56:1893)
[ 29.390264] NVRM: GPU 0000:01:00.0: rm_init_adapter failed, device minor number 0
[ 29.433187] NVRM: GPU 0000:01:00.0: RmInitAdapter failed! (0x61:0x56:1893)
[ 29.433351] NVRM: GPU 0000:01:00.0: rm_init_adapter failed, device minor number 0
[ 29.470100] NVRM: GPU 0000:01:00.0: RmInitAdapter failed! (0x61:0x56:1893)
[ 29.470267] NVRM: GPU 0000:01:00.0: rm_init_adapter failed, device minor number 0
[ 29.505669] NVRM: GPU 0000:01:00.0: RmInitAdapter failed! (0x61:0x56:1893)
[ 29.505827] NVRM: GPU 0000:01:00.0: rm_init_adapter failed, device minor number 0
[ 29.542207] NVRM: GPU 0000:01:00.0: RmInitAdapter failed! (0x61:0x56:1893)
[ 29.542367] NVRM: GPU 0000:01:00.0: rm_init_adapter failed, device minor number 0
[ 339.323619] NVRM: GPU 0000:01:00.0: RmInitAdapter failed! (0x61:0x56:1893)
[ 339.324295] NVRM: GPU 0000:01:00.0: rm_init_adapter failed, device minor number 0
Thank you
Hi,
Could you check the OS version in your environment first?
$ cat /etc/nv_tegra_release
If the system is 38.2.2, your custom kernel might be overwritten by our default kernel during the upgrade procedure.
To prevent this, please follow the document below to rename the kernel and update extlinux.conf file:
Thanks.
Hi,
Thank you for the response. Here is the content of nv_tegra_release:
R38 (release), REVISION: 2.2, GCID: 42205042, BOARD: generic, EABI: aarch64, DATE: Thu Sep 25 22:47:11 UTC 2025
KERNEL_VARIANT: oot
TARGET_USERSPACE_LIB_DIR=nvidia
TARGET_USERSPACE_LIB_DIR_PATH=usr/lib/aarch64-linux-gnu/nvidia
INSTALL_TYPE=
The nvidia-smi No devices found also happens before calling apt upgrade.
I have tried booting from a renamed custom image like suggested, and I have been using a custom .conf file when flashing (based on the jetson-agx-thor-devkit config), but the issue persists.
Probing further, upon unloading and re-loading the nvidia kernel modules:
[ 207.726441] NVRM: loading NVIDIA UNIX Open Kernel Module for aarch64 TempVersion Release Build (bugfix_main) (root@595d3405229e) Mon Nov 24 12:46:54 UTC 2025
[ 207.757935] nvidia-modeset: Loading NVIDIA UNIX Open Kernel Mode Setting Driver for aarch64 TempVersion Release Build (bugfix_main) (root@595d3405229e) Mon Nov 24 12:46:45 UTC 2025
[ 207.769236] [drm] [nvidia-drm] [GPU ID 0x00000100] Loading driver
[ 207.858797] nvidia 0000:01:00.0: Direct firmware load for nvidia/580.00/gsp_ga10x.bin failed with error -2
[ 207.858835] NVRM: RmFetchGspRmImages: No firmware image found
[ 207.858844] NVRM: GPU 0000:01:00.0: RmInitAdapter failed! (0x61:0x56:1893)
[ 207.859622] NVRM: GPU 0000:01:00.0: rm_init_adapter failed, device minor number 0
[ 207.860323] [drm:nv_drm_dev_load [nvidia_drm]] ERROR [nvidia-drm] [GPU ID 0x00000100] Failed to allocate NvKmsKapiDevice
[ 207.865837] [drm:nv_drm_register_drm_device [nvidia_drm]] ERROR [nvidia-drm] [GPU ID 0x00000100] Failed to load device
[ 207.876719] [drm] [nvidia-drm] [GPU ID 0x00020000] Loading driver
[ 208.152658] nvidia-modeset: WARNING: HW supports 8 heads. Limiting to 4 heads
[ 208.153141] nvidia-modeset: WARNING: GPU:0: The head configuration (0xff) is inconsistent with the number of heads (4)
[ 208.898018] NVRM: rpcRmApiControl_dce: NVRM_RPC_DCE: Failed RM ctrl call cmd:0x731341 result 0xffff: Failure: Generic Error [NV_ERR_GENERIC]
[ 209.924531] [drm] Initialized nvidia-drm 0.0.0 20160202 for 8808c00000.display on minor 1
[ 209.924551] [drm] [nvidia-drm] [GPU ID 0x00020000] Invalid framebuffer console info
[ 210.427958] nv_platform 8808c00000.display: [drm] Cannot find any crtc or sizes
It’s quite likely that this is the culprit:
[ 207.858797] nvidia 0000:01:00.0: Direct firmware load for nvidia/580.00/gsp_ga10x.bin failed with error -2
[ 207.858835] NVRM: RmFetchGspRmImages: No firmware image found
Could you point me to the step in the build process this image should’ve been built/installed?
Hi,
Please check the below comment to see if any step is missing when applying the custom kernel:
Please also note that the apply_binaries.sh command for Thor requires an extra --openrm flag:
https://docs.nvidia.com/jetson/archives/r38.2/DeveloperGuide/IN/QuickStart.html#to-flash-the-jetson-developer-kit-operating-software
sudo ./apply_binaries.sh --openrm
Thanks.
Hi,
The missing openrm flag was the culprit. This was one step I forgot to check when upgrading to Thor.
This is the output of nvidia-smi now:
±----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.00 Driver Version: 580.00 CUDA Version: 13.0 |
±----------------------------------------±-----------------------±---------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA Thor Off | 00000000:01:00.0 Off | N/A |
| N/A N/A N/A N/A / N/A | Not Supported | 0% Default |
| | | Disabled |
±----------------------------------------±-----------------------±---------------------+
±----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 2043 G /usr/lib/xorg/Xorg 0MiB |
±----------------------------------------------------------------------------------------+
The GPU also shows up in jtop. Thank you!