Intermittent missing GPU devfreq node on cold boot – Jetson AGX Orin – JetPack 7.2 (L4T 39.2)

43300371 · September 24, 2026, 3:34am

Hi NVIDIA team,

We are observing an intermittent issue on our Jetson AGX Orin Developer Kit after upgrading from JetPack 6.2 to JetPack 7.2. On some cold boots, the GPU devfreq node is not created, which causes nvpmodel, jetson_clocks, and jtop to fail. A simple reboot always restores normal operation.

System information:
$ uname -a
Linux agx-133 6.8.12-1021-tegra #1 SMP PREEMPT Mon Jun 1 13:25:46 PDT 2026 aarch64 aarch64 aarch64 GNU/Linux

$ cat /etc/nv_tegra_release
R39 (release), REVISION: 2.0, GCID: 45755727, BOARD: generic, EABI: aarch64, DATE: Mon Jun 1 09:28:48 PM UTC 2026
KERNEL_VARIANT: oot
TARGET_USERSPACE_LIB_DIR=nvidia
TARGET_USERSPACE_LIB_DIR_PATH=usr/lib/aarch64-linux-gnu/nvidia

$ sudo nvpmodel -q
NV Power Mode: MAXN
0
Problem description:

After switching from JetPack 6.2 to JetPack 7.2, we occasionally see the following on cold boot:

GPU devfreq node missing:
/sys/devices/platform/17000000.gpu/devfreq/17000000.gpu/ does not exist.
As a result, nvpmodel, jetson_clocks, and jtop fail in a chain.
reboot always fixes the problem.

Reproduction rate:

Cold boot: 1 failure out of 8 attempts.
reboot: 10 out of 10 successful.

Observed behavior when the problem occurs:
$ uptime
11:29:24 up 13 min, 2 users, load average: 0.00, 0.06, 0.08

$ jtop
The jtop.service is not active. Please run:
sudo jtop --install-service

$ sudo systemctl restart jtop.service

$ jtop
The jtop.service is not active. Please run:
sudo jtop --install-service

$ sudo systemctl status jtop.service
○ jtop.service - jtop service
Loaded: loaded (/etc/systemd/system/jtop.service; enabled; preset: enabled)
Active: inactive (dead) since Wed 2026-09-23 11:29:54 CST; 13s ago
Duration: 2.317s
Process: 8908 ExecStart=/usr/local/bin/jtop --force (code=exited, status=0/SUCCESS)
Main PID: 8908 (code=exited, status=0/SUCCESS)
CPU: 925ms

Sep 23 11:29:54 agx-133 jtop[8946]: self._engines_list = self.show()
Sep 23 11:29:54 agx-133 jtop[8946]: ^^^^^^^^^^^
Sep 23 11:29:54 agx-133 jtop[8946]: File “/usr/local/lib/python3.12/dist-packages/jtop/core/jetson_clocks.py”, line 522, in show
Sep 23 11:29:54 agx-133 jtop[8946]: lines = cmd(timeout=COMMAND_TIMEOUT)
Sep 23 11:29:54 agx-133 jtop[8946]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Sep 23 11:29:54 agx-133 jtop[8946]: File “/usr/local/lib/python3.12/dist-packages/jtop/core/command.py”, line 131, in call
Sep 23 11:29:54 agx-133 jtop[8946]: raise Command.CommandException(‘Error process:’, self.process.returncode)
Sep 23 11:29:54 agx-133 jtop[8946]: jtop.core.command.Command.CommandException: [errno:1] Error process:
Sep 23 11:29:54 agx-133 jtop[8908]: [INFO] jtop.service - Service closed
Sep 23 11:29:54 agx-133 systemd[1]: jtop.service: Deactivated successfully.

$ sudo jtop --install-service
[INFO] jtop.core.jetson_variables - Install jtop_env.sh variables
[INFO] jtop.core.jetson_variables - - Remove old /etc/profile.d/jtop_env.sh
[INFO] jtop.core.jetson_variables - - COPYING jtop_env.sh → /etc/profile.d/jtop_env.sh
[INFO] jtop.service - Install jtop.service
[INFO] jtop.service - - Remove old /etc/systemd/system/jtop.service
[INFO] jtop.service - - COPYING jtop.service → /etc/systemd/system/jtop.service
[INFO] jtop.service - - Reload all daemons
[INFO] jtop.service - - ENABLE jtop.service
[INFO] jtop.service - - START jtop.service
jtop.service installed and enabled

$ sudo systemctl status jtop.service
● jtop.service - jtop service
Loaded: loaded (/etc/systemd/system/jtop.service; enabled; preset: enabled)
Active: active (running) since Wed 2026-09-23 11:30:42 CST; 1s ago
Main PID: 9369 (jtop)
Tasks: 2 (limit: 74812)
Memory: 32.4M (peak: 49.4M)
CPU: 906ms
CGroup: /system.slice/jtop.service
└─9369 /usr/bin/python3 /usr/local/bin/jtop --force

Sep 23 11:30:44 agx-133 jtop[9428]: self.jetson_clocks.initialization(self.nvpmodel, data)
Sep 23 11:30:44 agx-133 jtop[9428]: File “/usr/local/lib/python3.12/dist-packages/jtop/core/jetson_clocks.py”, line 370, in initialization
Sep 23 11:30:44 agx-133 jtop[9428]: self._engines_list = self.show()
Sep 23 11:30:44 agx-133 jtop[9428]: ^^^^^^^^^^^
Sep 23 11:30:44 agx-133 jtop[9428]: File “/usr/local/lib/python3.12/dist-packages/jtop/core/jetson_clocks.py”, line 522, in show
Sep 23 11:30:44 agx-133 jtop[9428]: lines = cmd(timeout=COMMAND_TIMEOUT)
Sep 23 11:30:44 agx-133 jtop[9428]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Sep 23 11:30:44 agx-133 jtop[9428]: File “/usr/local/lib/python3.12/dist-packages/jtop/core/command.py”, line 131, in call
Sep 23 11:30:44 agx-133 jtop[9428]: raise Command.CommandException(‘Error process:’, self.process.returncode)
Sep 23 11:30:44 agx-133 jtop[9428]: jtop.core.command.Command.CommandException: [errno:1] Error process:

$ jtop
The jtop.service is not active. Please run:
sudo jtop --install-service

Root cause investigation:

We found that on the affected boot, the following directory is not created:
$ ls /sys/devices/platform/17000000.gpu/devfreq/17000000.gpu/
ls: cannot access ‘/sys/devices/platform/17000000.gpu/devfreq/17000000.gpu/’: No such file or directory
And nvpmodel reports:
$ sudo nvpmodel -q
NV Power Mode: MAXN
0
NVPM ERROR: Error opening /sys/devices/platform/17000000.gpu/devfreq_dev/min_freq: 2
NVPM ERROR: failed to read PARAM GPU: ARG MIN_FREQ: PATH /sys/devices/platform/17000000.gpu/devfreq_dev/min_freq
NVPM ERROR: Error opening /sys/devices/platform/17000000.gpu/devfreq_dev/max_freq: 2
NVPM ERROR: failed to read PARAM GPU: ARG MAX_FREQ: PATH /sys/devices/platform/17000000.gpu/devfreq_dev/max_freq

Additional logs:
$ sudo jetson_clocks --show
SOC family:tegra234 Machine:NVIDIA Jetson AGX Orin Developer Kit
Online CPUs: 0-11, Offline CPUs:
cpu0: Governor=schedutil MinFreq=2201600 MaxFreq=2201600 CurrentFreq=2201600 IdleStates: WFI=0 c7=0
cpu1: Governor=schedutil MinFreq=2201600 MaxFreq=2201600 CurrentFreq=2201600 IdleStates: WFI=0 c7=0
cpu2: Governor=schedutil MinFreq=2201600 MaxFreq=2201600 CurrentFreq=2201600 IdleStates: WFI=0 c7=0
cpu3: Governor=schedutil MinFreq=2201600 MaxFreq=2201600 CurrentFreq=2201600 IdleStates: WFI=0 c7=0
cpu4: Governor=schedutil MinFreq=2201600 MaxFreq=2201600 CurrentFreq=2201600 IdleStates: WFI=0 c7=0
cpu5: Governor=schedutil MinFreq=2201600 MaxFreq=2201600 CurrentFreq=2201600 IdleStates: WFI=0 c7=0
cpu6: Governor=schedutil MinFreq=2201600 MaxFreq=2201600 CurrentFreq=2201600 IdleStates: WFI=0 c7=0
cpu7: Governor=schedutil MinFreq=2201600 MaxFreq=2201600 CurrentFreq=2201600 IdleStates: WFI=0 c7=0
cpu8: Governor=schedutil MinFreq=2201600 MaxFreq=2201600 CurrentFreq=2201600 IdleStates: WFI=0 c7=0
cpu9: Governor=schedutil MinFreq=2201600 MaxFreq=2201600 CurrentFreq=2201600 IdleStates: WFI=0 c7=0
cpu10: Governor=schedutil MinFreq=2201600 MaxFreq=2201600 CurrentFreq=2201600 IdleStates: WFI=0 c7=0
cpu11: Governor=schedutil MinFreq=2201600 MaxFreq=2201600 CurrentFreq=2201600 IdleStates: WFI=0 c7=0
/usr/bin/jetson_clocks: line 552: echo: write error: Invalid argument
/usr/bin/jetson_clocks: line 558: echo: write error: Invalid argument
Error! GPU frequency scaling not supported!

Questions:

Is this a known issue in JetPack 7.2 (L4T 39.2)?
Has it been fixed in JetPack 7.2.1?
Is there any workaround to avoid the failure of nvpmodel, jetson_clocks, and jtop when the GPU devfreq node is missing on cold boot?

Any help or confirmation would be greatly appreciated.

Thanks.

c.smith · October 1, 2026, 5:01am

Seeing the same issue on a Jetson Orin NX 16GB on JetPack 7.2.1 (R39.2.1), so to answer question 2: it is not fixed in 7.2.1. It’s intermittent for me too (2 confirmed failures in ~37 logged boots), and a clean sudo reboot always fixes it.

The missing devfreq node looks like a downstream effect. In my dmesg, the GPU never finishes powering on because its ACR firmware fails to load at boot with error -40 (ELOOP, “too many levels of symbolic links”), even though the firmware file is intact. My failed boot also followed an unclean shutdown (ext4 journal recovery at mount), which fits your cold-boot-only pattern.

@43300371, if you get another failed cold boot, could you check whether you see the same first error?

sudo dmesg | grep -E "acr-gsp|error -40|ACR bootstrap"

Full details and raw logs below.

Setup

Jetson Orin NX 16GB on the Orin NX Engineering Reference Developer Kit
JetPack 7.2.1, L4T R39.2.1 (GCID 46758480), kernel 6.8.12-1021-tegra
Driver 595.78, CUDA 13.2
Power mode 15W, jetson_clocks running
Rootfs on a 2 TB NVMe SSD (/dev/nvme0n1p1, ext4), booted via initrd. Drive: Crucial P310 2TB (CT2000P310SSD2), firmware V8CR001
Out-of-tree modules also loaded: Morse Micro dot11ah/morse_usb, r8168
Kernel cmdline:

root=PARTUUID=34118af0-a548-43f2-83e8-df9dd9cf05aa rw rootwait rootfstype=ext4 mminit_loglevel=4 console=ttyTCU0,115200 firmware_class.path=/etc/firmware fbcon=map:0 video=efifb:off console=tty0 efi_pstore.pstore_disable=1 pstore.backend=ramoops efi=runtime pci=pcie_bus_perf nvme.use_threaded_interrupts=1 swiotlb=2048 pcie_aspm=off bl_prof_dataptr=2031616@0x467210000 bl_prof_ro_ptr=65536@0x467200000

Symptom

On a failed boot, the GPU never comes up and stays down for the whole boot:

nvidia-smi → libnvrm_gpu.so: NvRmGpuLibOpen failed, error=4 / No devices were found
tegrastats shows GR3D_FREQ 0% with no frequency
jtop shows “NVIDIA GPU NOT DETECTED/AVAILABLE”, and the gpu temperature sensor shows as offline
/dev/nvgpu/igpu0/ contains only power
Module and permissions are fine: nvgpu is loaded, and the user is in video and render

A clean sudo reboot brings it back (nvidia-smi shows Orin (nvgpu), and jtop shows GPU at 918 MHz).

Confirmed failures: Sep 18 (boot cb87fcf1…) and Sep 29 (boot 5331b87e…). Both show the same 9 acr-gsp ... error -40 lines.

First error in dmesg (failed boot, Sep 29)

The acr-gsp firmware load fails with -40 (ELOOP) on every search path:

[   43.392385] nvgpu: 17000000.gpu                    ga10b_init_hal:2237 [INFO]  Semaphore-based GP_GET tracking is not intended for GA10B production drivers
[   43.423510] gk20a 17000000.gpu: loading /lib/firmware/updates/6.8.12-1021-tegra/nvidia/ga10b/acr-gsp.data.encrypt.bin.prod failed with error -40
[   43.423898] gk20a 17000000.gpu: loading /lib/firmware/updates/nvidia/ga10b/acr-gsp.data.encrypt.bin.prod failed with error -40
[   43.424218] gk20a 17000000.gpu: loading /lib/firmware/6.8.12-1021-tegra/nvidia/ga10b/acr-gsp.data.encrypt.bin.prod failed with error -40
[   43.424222] gk20a 17000000.gpu: loading /lib/firmware/nvidia/ga10b/acr-gsp.data.encrypt.bin.prod failed with error -40
[   43.424235] gk20a 17000000.gpu: loading /lib/firmware/updates/6.8.12-1021-tegra/ga10b/acr-gsp.data.encrypt.bin.prod failed with error -40
[   43.427348] gk20a 17000000.gpu: loading /lib/firmware/updates/ga10b/acr-gsp.data.encrypt.bin.prod failed with error -40
[   43.427698] gk20a 17000000.gpu: loading /lib/firmware/6.8.12-1021-tegra/ga10b/acr-gsp.data.encrypt.bin.prod failed with error -40
[   43.429006] gk20a 17000000.gpu: loading /lib/firmware/ga10b/acr-gsp.data.encrypt.bin.prod failed with error -40
[   43.438033] gk20a 17000000.gpu: Direct firmware load for ga10b/acr-gsp.data.encrypt.bin.prod failed with error -40
[   43.447975] nvgpu: 17000000.gpu       ga10b_load_riscv_acr_ucodes:454  [ERR]  acr-gsp.data.encrypt.bin.prod ucode get fail for ga10b
[   43.459912] nvgpu: 17000000.gpu nvgpu_acr_bootstrap_hs_ucode_riscv:481  [ERR]  RISCV ucode loading failed
[   43.469601] nvgpu: 17000000.gpu            ga10b_bootstrap_hs_acr:53   [ERR]  ACR bootstrap failed
[   43.478585] nvgpu: 17000000.gpu        nvgpu_acr_bootstrap_hs_acr:164  [ERR]  ACR bootstrap failed
[   43.487619] nvgpu: 17000000.gpu       nvgpu_acr_construct_execute:187  [ERR]  Bootstrap HS ACR failed
[   43.496961] nvgpu: 17000000.gpu            nvgpu_finalize_poweron:1287 [ERR]  Failed initialization for: g->ops.acr.acr_construct_execute

Immediately after, the display RM’s DCE RPCs time out:

[   43.516685] [drm] [nvidia-drm] [GPU ID 0xfffffffe] Loading driver
[   43.798362] NVRM: GPU0 rpcRmApiControl_dce: NVRM_RPC_DCE: Failed RM ctrl call cmd:0x731341 result 0x65: Call timed out [NV_ERR_TIMEOUT]
   ... (16 times, 43.798 → 43.962)
[   43.963694] NVRM: GPU0 rpcRmApiControl_dce: NVRM_RPC_DCE: Failed RM ctrl call cmd:0x730282 result 0x1f: Invalid argument to call [NV_ERR_INVALID_ARGUMENT]
[   44.099439] [drm] [nvidia-drm] [GPU ID 0xfffffffe] Invalid framebuffer console info

After that, every attempt to use the GPU for the rest of the boot (each nvidia-smi, CUDA init, jtop, etc.) fails the same way. The firmware load is never retried:

[   44.452481] nvgpu: 17000000.gpu ga10b_acr_patch_wpr_info_to_ucode:215  [ERR]  invalid mem acr_falcon2_sysmem_desc
[   44.452803] nvgpu: 17000000.gpu nvgpu_acr_bootstrap_hs_ucode_riscv:475  [ERR]  RISCV ucode patch wpr info failed
[   44.453078] nvgpu: 17000000.gpu            ga10b_bootstrap_hs_acr:53   [ERR]  ACR bootstrap failed
[   44.453327] nvgpu: 17000000.gpu        nvgpu_acr_bootstrap_hs_acr:164  [ERR]  ACR bootstrap failed
[   44.453581] nvgpu: 17000000.gpu       nvgpu_acr_construct_execute:187  [ERR]  Bootstrap HS ACR failed
[   44.453838] nvgpu: 17000000.gpu            nvgpu_finalize_poweron:1287 [ERR]  Failed initialization for: g->ops.acr.acr_construct_execute
[   46.437836] nvgpu: 17000000.gpu                 gk20a_power_write:98   [ERR]  power_node_write failed at busy
   ... (repeats on every access, still repeating at 184646 s uptime)

The firmware file is intact

Checked while the failure was happening:

$ namei -l /lib/firmware/nvidia/ga10b/acr-gsp.data.encrypt.bin.prod
f: /lib/firmware/nvidia/ga10b/acr-gsp.data.encrypt.bin.prod
drwxr-xr-x root root /
lrwxrwxrwx root root lib -> usr/lib
drwxr-xr-x root root   usr
drwxr-xr-x root root   lib
drwxr-xr-x root root firmware
drwxr-xr-x root root nvidia
drwxr-xr-x root root ga10b
-rw-r--r-- root root acr-gsp.data.encrypt.bin.prod

$ stat -c '%n  %z' /lib/firmware/nvidia/ga10b/acr-gsp.data.encrypt.bin.prod
/lib/firmware/nvidia/ga10b/acr-gsp.data.encrypt.bin.prod  1970-01-01 10:02:06.872000000 +1000

It’s a regular file, unchanged since flashing, and the only symlink in the path is the standard /lib -> usr/lib. No packages were reinstalled; a plain reboot fixes it.

The key detail: paths that don’t exist also returned -40. /lib/firmware/ga10b/... and /lib/firmware/updates/... should return -2 (ENOENT), but returned -40 instead. So the lookup failed in the part all 8 paths share (/ or /lib), not at the firmware file. For a brief moment at boot, the kernel firmware loader couldn’t resolve /lib.

Boot timing: failed vs good

The failed boot was about 30 s slower to reach the GPU probe:

	Good boot (Oct 1)	Failed boot (Sep 29)
r8152 firmware load	~8 s	~41 s
`ga10b_init_hal` (nvgpu probe)	~11 s	~43 s

What filled the gap in the failed boot (dmesg -T, wall-clock times):

[Tue Sep 29 10:38:27 2026] tegra-pmc c360000.pmc: sync_state() pending due to 4-003c
                           (~20 s of silence)
[Tue Sep 29 10:38:47 2026] nvme nvme0: I/O tag 998 (83e6) QID 3 timeout, completion polled
[Tue Sep 29 10:38:48 2026] EXT4-fs (nvme0n1p1): 2 orphan inodes deleted
[Tue Sep 29 10:38:48 2026] EXT4-fs (nvme0n1p1): recovery complete
[Tue Sep 29 10:38:48 2026] EXT4-fs (nvme0n1p1): mounted filesystem a3c7bb2d-7b14-41a0-a7c6-45d7e15df924 r/w with ordered data mode. Quota mode: none.
[Tue Sep 29 10:38:48 2026] Rootfs mounted over PARTUUID=34118af0-a548-43f2-83e8-df9dd9cf05aa
[Tue Sep 29 10:38:48 2026] Switching from initrd to actual rootfs
                           ... ~5 s later: nvgpu probe → acr-gsp load fails with -40

In this boot:

An NVMe I/O command timed out. The drive had completed it, but the completion interrupt was missed and the kernel only found out by polling after the timeout.
ext4 replayed its journal, so the previous shutdown was unclean.
Both happened right before the initrd → rootfs switch, and the GPU firmware request failed about 5 s later.

CPU / power mode difference (possibly related)

jtop on the failed boot showed CPU cores 5–8 OFF and the active cores at 1.4 GHz. On the good boot, all 8 cores are online at 2.0 GHz. Both report NV Power[2]: 15W with jetson_clocks running. The original report in this thread says nvpmodel and jetson_clocks fail when the GPU doesn’t come up, so the power mode may also be applied incorrectly on failed boots. The good-boot nvpmodel -q --verbose output is in system_info.txt. I don’t have the equivalent from the failed boot.

Hypothesis (not confirmed)

The acr-gsp firmware request happens during or close to the initrd → rootfs transition, at a moment when path resolution through /lib returns ELOOP. The pattern in this thread (fails only on cold boot, never on reboot) fits this: a cold boot after an unclean shutdown triggers ext4 journal recovery and slower NVMe bring-up, which shifts the timing of the initrd → rootfs switch enough to hit that window. A clean reboot doesn’t recover a journal and hasn’t failed for me or for the original poster.

I couldn’t correlate this across all older boots because some of my journal history has been rotated out. I’ll capture the next failure in full, including how the previous boot ended.

Questions

Is this a known or tracked issue on R39.2.x for Orin? It looks the same as the sporadic NvRmGpuLibOpen failed, error=4 reports on Orin Nano with JetPack 7.2.
Is an ELOOP result from the firmware loader expected if the request lands during or just after the initrd switch? Should nvgpu be loaded from the initrd, or held back until the real rootfs is fully mounted?
Could nvgpu retry the firmware load on later power-on attempts instead of failing for the whole boot? Is there a supported way to recover without rebooting, for example unbind/bind of 17000000.gpu on the gk20a driver?
Is the nvme ... timeout, completion polled message known on Orin NX with nvme.use_threaded_interrupts=1 / pcie_aspm=off?
In 15W mode on Orin NX 16GB, which CPU configuration is expected: 4 cores at 1.4 GHz, or 8 cores at 2.0 GHz? Is nvpmodel expected to fail to apply when the GPU devfreq node is missing?

Attached: `gpu_bug_logs_2026-10-01.zip` (raw, untrimmed logs)

Failed boot, Sep 29

failed_boot_sep29_dmesg_full.txt: full dmesg -T, saved before rebooting
failed_boot_sep29_dmesg_gpu_grep.txt: dmesg | grep -iE "gpu|nvgpu|ga10b|dce|error" (monotonic timestamps)
failed_boot_sep29_journal_kernel.txt / failed_boot_sep29_journal_full.txt: journal for that boot (early-boot entries were rotated out, so use the dmesg file for those)
diagnostics.txt: nvidia-smi, tegrastats, lsmod, /dev/nvgpu, groups, namei, stat captured during the failure

Failed boot, Sep 18

failed_boot_sep18_journal_kernel.txt / failed_boot_sep18_journal_full.txt

Good boot, Oct 1 (after sudo reboot)

good_boot_oct1_dmesg_full.txt / good_boot_oct1_dmesg_full_walltime.txt
good_boot_oct1_dmesg_gpu_grep.txt

System

system_info.txt: release, kernel, NVMe model/firmware, cmdline, extlinux.conf, initrd, firmware path, nvpmodel, nvidia-smi, lsmod, L4T package versions
boot_list.txt: journalctl --list-boots

gpu_bug_logs_2026-10-01.zip (160.3 KB)

whitesscott · October 1, 2026, 6:59am

You could try

sudo apt update
sudo apt install --reinstall nvidia-l4t-firmware nvidia-l4t-core nvidia-l4t-kernel 

sudo nv-update-initrd
sudo reboot

c.smith · October 2, 2026, 12:19am

This has occurred on multiple Jetsons with fresh installs. As well as reinstalling kernel and firmware multiple times. I have only seen it on on Jetpack 7.2, and only a reboot fixes it.

mijnbureauklok · October 3, 2026, 7:57am

Claude found the issue on my Orin Nano, here is its write-up:

Root cause found, with a deterministic repro and a workaround

Same symptoms on an Orin Nano 8GB (L4T 39.2, kernel 6.8.12-1021-tegra): about 1 in 25 boots, warm reboots included, acr-gsp.*.encrypt.bin.prod failed with error -40, then ACR bootstrap failed and no GPU until the next reboot. It is not the initrd switch (the GPU probes ~6 s after it) and not a bad firmware file.

Cause: nvgpu loads its firmware during the first GPU power-on. At boot two services race to do that:

nvpmodel writes /dev/nvhost-power-gpu: works
nvidia-cdi-refresh.service (nvidia-ctk cdi generate, via NVML) opens /dev/nvgpu/igpu0/ctrl: fails

When the open() wins, the whole power-on runs inside path_openat(). Every request_firmware() lookup is then a nested path walk that shares the outer open’s 40-symlink budget (__set_nameidata() inherits total_link_count, restore_nameidata() writes it back). Every candidate firmware path crosses the /lib -> usr/lib symlink, so after ~10 firmware files the budget is used up and the ACR files get ELOOP on every path.

Stack from a failing boot (printk stacktrace trigger), process nvidia-ctk:

_request_firmware < request_firmware_direct < nvgpu_request_firmware
< nvgpu_acr_bootstrap_hs_ucode_riscv < ... < nvgpu_finalize_poweron
< gk20a_pm_finalize_poweron < ... < gk20a_busy < gk20a_ctrl_dev_open
< chrdev_open < vfs_open < path_openat < do_filp_open < __arm64_sys_openat

On healthy boots the first power-on comes from gk20a_power_write (a write(), no outer path walk).

Repro: make an open() the first GPU access, for example a oneshot unit (DefaultDependencies=no, WantedBy=sysinit.target) running

bash -c 'until [ -e /dev/nvgpu/igpu0/ctrl ]; do sleep 0.01; done; exec 3<>/dev/nvgpu/igpu0/ctrl'

GPU dead on 2 of 2 boots.

Workaround: search the real firmware directory first, so the lookups follow no symlinks:

# /etc/systemd/system/firmware-path.service
[Unit]
DefaultDependencies=no
Before=systemd-udev-trigger.service systemd-modules-load.service nv-load-display-modules.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/sh -c 'printf /usr/lib/firmware > /sys/module/firmware_class/parameters/path'

[Install]
WantedBy=sysinit.target

Same repro with this unit: 3 of 3 boots fine. It is also why the /etc/firmware copy from jetson-ai-lab issue #427 helps: the first path tried becomes a real directory.

Suggested fix: load the firmware at probe time instead of from the device open() path, or ship firmware_class.path pointing at a directory that is not behind a symlink.

43300371 · October 10, 2026, 4:11am

yes. I see the same first error. My AGX Orin is 64GB

The full details below.
ai@ai:~$ sudo dmesg | grep [ 19.460750] gk20a 17000000.gpu: [ 19.474510] gk20a 17000000.gpu: [ 19.487392] gk20a 17000000.gpu: [ 19.500806] gk20a 17000000.gpu: [ 19.512155] gk20a 17000000.gpu: [ 19.524924] gk20a 17000000.gpu: [ 19.536124] gk20a 17000000.gpu: [ 19.548166] gk20a 17000000.gpu: [ 19.558582] gk20a 17000000.gpu: [ 19.569267] nvgpu: 17000000.gpu [ 19.591447] nvgpu: 17000000.gpu [ 19.600687] nvgpu: 17000000.gpu [ 19.672412] nvgpu: 17000000.gpu [ 19.681656] nvgpu: 17000000.gpu [ 19.849280] nvgpu: 17000000.gpu [ 19.849284] nvgpu: 17000000.gpu [ 19.917713] nvgpu: 17000000.gpu [ 19.918035] nvgpu: 17000000.gpu [ 20.224200] nvgpu: 17000000.gpu [ 20.224205] nvgpu: 17000000.gpu [ 21.225036] nvgpu: 17000000.gpu [ 21.234312] nvgpu: 17000000.gpu [ 23.191641] nvgpu: 17000000.gpu [ 23.191644] nvgpu: 17000000.gpu [ 23.743889] nvgpu: 17000000.gpu [ 23.753142] nvgpu: 17000000.gpu [ 23.846979] nvgpu: 17000000.gpu [ 23.856221] nvgpu: 17000000.gpu [ 23.969904] nvgpu: 17000000.gpu [ 23.979167] nvgpu: 17000000.gpu [ 24.084207] nvgpu: 17000000.gpu [ 24.093477] nvgpu: 17000000.gpu [ 24.183559] nvgpu: 17000000.gpu [ 24.192815] nvgpu: 17000000.gpu [ 24.293608] nvgpu: 17000000.gpu [ 24.293611] nvgpu: 17000000.gpu [ 24.396727] nvgpu: 17000000.gpu [ 24.405974] nvgpu: 17000000.gpu [ 24.508975] nvgpu: 17000000.gpu [ 24.518244] nvgpu: 17000000.gpu [ 24.828130] nvgpu: 17000000.gpu [ 24.837399] nvgpu: 17000000.gpu [ 25.421522] nvgpu: 17000000.gpu [ 25.430763] nvgpu: 17000000.gpu [ 28.751348] nvgpu: 17000000.gpu [ 28.760659] nvgpu: 17000000.gpu [ 137.381372] nvgpu: 17000000.gpu [ 137.390687] nvgpu: 17000000.gpu [ 137.522409] nvgpu: 17000000.gpu [ 137.531688] nvgpu: 17000000.gpu [ 137.660033] nvgpu: 17000000.gpu [ 137.669276] nvgpu: 17000000.gpu [ 139.082278] nvgpu: 17000000.gpu [ 139.091581] nvgpu: 17000000.gpu [ 139.203563] nvgpu: 17000000.gpu [ 139.212807] nvgpu: 17000000.gpu [ 140.042469] nvgpu: 17000000.gpu [ 140.051730] nvgpu: 17000000.gpu [ 140.140178] nvgpu: 17000000.gpu [ 140.149439] nvgpu: 17000000.gpu -E “acr-gsp|error -40|ACR bootstrap”
loading /lib/firmware/updates/6.8.12-1021-tegra/nvidia/ga10b/acr-gsp.data.encrypt.bin.prod failed with error -40
loading /lib/firmware/updates/nvidia/ga10b/acr-gsp.data.encrypt.bin.prod failed with error -40
loading /lib/firmware/6.8.12-1021-tegra/nvidia/ga10b/acr-gsp.data.encrypt.bin.prod failed with error -40
loading /lib/firmware/nvidia/ga10b/acr-gsp.data.encrypt.bin.prod failed with error -40
loading /lib/firmware/updates/6.8.12-1021-tegra/ga10b/acr-gsp.data.encrypt.bin.prod failed with error -40
loading /lib/firmware/updates/ga10b/acr-gsp.data.encrypt.bin.prod failed with error -40
loading /lib/firmware/6.8.12-1021-tegra/ga10b/acr-gsp.data.encrypt.bin.prod failed with error -40
loading /lib/firmware/ga10b/acr-gsp.data.encrypt.bin.prod failed with error -40
Direct firmware load for ga10b/acr-gsp.data.encrypt.bin.prod failed with error -40
ga10b_load_riscv_acr_ucodes:454 [ERR] acr-gsp.data.encrypt.bin.prod ucode get fail for ga10b
ga10b_bootstrap_hs_acr:53 [ERR] ACR bootstrap failed
nvgpu_acr_bootstrap_hs_acr:164 [ERR] ACR bootstrap failed
ga10b_bootstrap_hs_acr:53 [ERR] ACR bootstrap failed
nvgpu_acr_bootstrap_hs_acr:164 [ERR] ACR bootstrap failed
ga10b_bootstrap_hs_acr:53 [ERR] ACR bootstrap failed
nvgpu_acr_bootstrap_hs_acr:164 [ERR] ACR bootstrap failed
ga10b_bootstrap_hs_acr:53 [ERR] ACR bootstrap failed
nvgpu_acr_bootstrap_hs_acr:164 [ERR] ACR bootstrap failed
ga10b_bootstrap_hs_acr:53 [ERR] ACR bootstrap failed
nvgpu_acr_bootstrap_hs_acr:164 [ERR] ACR bootstrap failed
ga10b_bootstrap_hs_acr:53 [ERR] ACR bootstrap failed
nvgpu_acr_bootstrap_hs_acr:164 [ERR] ACR bootstrap failed
ga10b_bootstrap_hs_acr:53 [ERR] ACR bootstrap failed
nvgpu_acr_bootstrap_hs_acr:164 [ERR] ACR bootstrap failed
ga10b_bootstrap_hs_acr:53 [ERR] ACR bootstrap failed
nvgpu_acr_bootstrap_hs_acr:164 [ERR] ACR bootstrap failed
ga10b_bootstrap_hs_acr:53 [ERR] ACR bootstrap failed
nvgpu_acr_bootstrap_hs_acr:164 [ERR] ACR bootstrap failed
ga10b_bootstrap_hs_acr:53 [ERR] ACR bootstrap failed
nvgpu_acr_bootstrap_hs_acr:164 [ERR] ACR bootstrap failed
ga10b_bootstrap_hs_acr:53 [ERR] ACR bootstrap failed
nvgpu_acr_bootstrap_hs_acr:164 [ERR] ACR bootstrap failed
ga10b_bootstrap_hs_acr:53 [ERR] ACR bootstrap failed
nvgpu_acr_bootstrap_hs_acr:164 [ERR] ACR bootstrap failed
ga10b_bootstrap_hs_acr:53 [ERR] ACR bootstrap failed
nvgpu_acr_bootstrap_hs_acr:164 [ERR] ACR bootstrap failed
ga10b_bootstrap_hs_acr:53 [ERR] ACR bootstrap failed
nvgpu_acr_bootstrap_hs_acr:164 [ERR] ACR bootstrap failed
ga10b_bootstrap_hs_acr:53 [ERR] ACR bootstrap failed
nvgpu_acr_bootstrap_hs_acr:164 [ERR] ACR bootstrap failed
ga10b_bootstrap_hs_acr:53 [ERR] ACR bootstrap failed
nvgpu_acr_bootstrap_hs_acr:164 [ERR] ACR bootstrap failed
ga10b_bootstrap_hs_acr:53 [ERR] ACR bootstrap failed
nvgpu_acr_bootstrap_hs_acr:164 [ERR] ACR bootstrap failed
ga10b_bootstrap_hs_acr:53 [ERR] ACR bootstrap failed
nvgpu_acr_bootstrap_hs_acr:164 [ERR] ACR bootstrap failed
ga10b_bootstrap_hs_acr:53 [ERR] ACR bootstrap failed
nvgpu_acr_bootstrap_hs_acr:164 [ERR] ACR bootstrap failed
ga10b_bootstrap_hs_acr:53 [ERR] ACR bootstrap failed
nvgpu_acr_bootstrap_hs_acr:164 [ERR] ACR bootstrap failed
ga10b_bootstrap_hs_acr:53 [ERR] ACR bootstrap failed
nvgpu_acr_bootstrap_hs_acr:164 [ERR] ACR bootstrap failed
ga10b_bootstrap_hs_acr:53 [ERR] ACR bootstrap failed
nvgpu_acr_bootstrap_hs_acr:164 [ERR] ACR bootstrap failed
ga10b_bootstrap_hs_acr:53 [ERR] ACR bootstrap failed
nvgpu_acr_bootstrap_hs_acr:164 [ERR] ACR bootstrap failed
ga10b_bootstrap_hs_acr:53 [ERR] ACR bootstrap failed
nvgpu_acr_bootstrap_hs_acr:164 [ERR] ACR bootstrap failed
ga10b_bootstrap_hs_acr:53 [ERR] ACR bootstrap failed
nvgpu_acr_bootstrap_hs_acr:164 [ERR] ACR bootstrap failed

43300371 · October 10, 2026, 12:41pm

I can confirm the same issue on a Jetson AGX Orin Developer Kit running JetPack 7.2 (L4T 39.2) with kernel 6.8.12-1021-tegra.

On cold boot, the GPU devfreq node was occasionally missing:

/sys/devices/platform/17000000.gpu/devfreq/17000000.gpu/ not found

This caused nvpmodel, jetson_clocks, and jtop to fail. A simple reboot always restored normal operation. Reproduction rate was about 1 in 8 cold boots, while 10 warm reboots all succeeded.

The dmesg output matched what others have reported:

gk20a 17000000.gpu: loading /lib/firmware/…/ga10b/acr-gsp.data.encrypt.bin.prod failed with error -40
nvgpu: 17000000.gpu ga10b_load_riscv_acr_ucodes:454 [ERR] acr-gsp.data.encrypt.bin.prod ucode get fail for ga10b
nvgpu: 17000000.gpu ga10b_bootstrap_hs_acr:53 [ERR] ACR bootstrap failed

I then applied the workaround suggested by mijnbureauklok:

/etc/systemd/system/firmware-path.service

[Unit]
DefaultDependencies=no
Before=systemd-udev-trigger.service systemd-modules-load.service nv-load-display-modules.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/sh -c ‘printf /usr/lib/firmware > /sys/module/firmware_class/parameters/path’

[Install]
WantedBy=sysinit.target

After enabling this service, I performed 40 cold boots and the problem did not reappear even once.

This strongly supports the root cause analysis: the nested open() path from nvidia-cdi-refresh.service exhausts the symlink budget, causing ELOOP (-40) on the ACR firmware paths that go through /lib -> /usr/lib.

Could NVIDIA confirm whether this root cause is acknowledged, and whether a proper fix is planned for an upcoming JetPack release? The workaround is effective but it would be good to have an official fix so users don’t have to rely on a custom systemd unit.

c.smith · October 12, 2026, 10:52pm

Thank you. I have run this workaround for a few days now. 67 cold boots and counting without error.

Topic		Replies	Views
Jetson Orin AGX not booting after cuda install Jetson AGX Orin jetpack , boot , reflash , cuda	9	1289	November 10, 2023
AGX Orin developer kit wont boot Jetson AGX Orin boot	7	330	August 7, 2024
After kernel minimization on Jetson Nano is not starting GPU Jetson Nano kernel , gpu	18	1386	April 19, 2024
Gpu not work Jetson AGX Orin gpu	20	656	August 6, 2025
Orin fails to boot after 5.1 update Jetson AGX Orin boot , reflash	54	4589	May 2, 2023
Jetson AGX orin Boot Stuck with "nvme0n1p1 not found" error Jetson AGX Orin boot	23	552	August 12, 2025
Jetson AGX Orin Bootup Failed Jetson AGX Orin boot	18	1680	December 5, 2023
Jetson Orin Nano Developer Kit Super (VFS failed, DCE errors, GPU unstable under load) - RMA request Jetson Orin Nano rma	9	382	July 17, 2026
CPU:0, Error: cbb-fabric@0x13a00000, irq=32 Jetson AGX Orin kernel	18	2129	July 17, 2024
The Jetson Orin Nano is using JetPack 39.2. Upon boot-up, the following error message appears Jetson Orin Nano boot , nvbugs	24	384	August 7, 2026

Intermittent missing GPU devfreq node on cold boot – Jetson AGX Orin – JetPack 7.2 (L4T 39.2)

Setup

Symptom

First error in dmesg (failed boot, Sep 29)

The firmware file is intact

Boot timing: failed vs good

CPU / power mode difference (possibly related)

Hypothesis (not confirmed)

Questions

Attached: gpu_bug_logs_2026-10-01.zip (raw, untrimmed logs)

/etc/systemd/system/firmware-path.service

Related topics

Attached: `gpu_bug_logs_2026-10-01.zip` (raw, untrimmed logs)