Jetson Thor (R38.4/JP7.1): devfreq ↔ rmapiLockAcquire deadlock wedges all GPU tasks in D-state

Robotics & Edge Computing Jetson Systems Jetson Thor
, ,
1

We’re running a fleet of Jetson AGX Thor modules on CTI Gauntlet (AGX301-class) carriers and are hitting a reproducible driver-level deadlock: the devfreq GPU load-sampling path appears to deadlock against the RM API lock, wedging every GPU-touching process in uninterruptible (D) state. The only recovery is a hard reboot. Posting in case it’s already tracked, and to see if there’s a fix or a fuller workaround.

Platform / versions

Module:   Jetson AGX Thor
Carrier:  CTI Gauntlet + Thor AGX
L4T/BSP:  R38 (release), REVISION 4.0  (JetPack 7.1), OpenRM (INSTALL_TYPE=openrm)
Kernel:   6.8.12-tegra #1 SMP PREEMPT
Rootfs:   Ubuntu 24.04
GCID:     43443517
Access:   headless / remote only (SSH, Tailscale, RustDesk on a virtual dummy display); no physical display
Workload: CUDA application — no camera/Argus, no display output

We cannot move to R39.2 / JetPack 7.2 yet, and the R39.2 release notes do not list this issue as fixed.

Symptom

Under normal CUDA load the system intermittently enters a state where every GPU-touching process (our app, the compositor, RustDesk display access) blocks in D-state. The hung-task detector reports blocked tasks at the 120s threshold and re-reports at ~241s / ~362s / ~483s as the wedge persists. Observed on multiple units.

Deadlock chain (as we understand it)

The blocked kworker is the devfreq monitor, stuck acquiring the RM API write lock:

devfreq_monitor (kworker, D-state)
  -> [governor_pod_scaling / devfreq load sampling]
  -> rm_pmu_perfmon_get_load
  -> rmapiLockAcquire
  -> os_acquire_rwlock_write
     (never acquired -> task wedged)

Once devfreq_wq is wedged, other GPU-touching tasks queue behind the same lock and also go D-state.

Earlier in the same boot (~14.4s) we also see an RM control failure that may indicate RM/DCE instability preceding the hang:

NVRM: rpcRmApiControl_dce: ... NV_ERR_GENERIC

This looks like the same lock-acquisition chain reported in other Thor R38/JP7 threads here on the forums, where setting devfreq governors to performance did not eliminate the deadlock.

Questions

  1. Is this devfreq ↔ rmapiLockAcquire deadlock a known issue, and is there a bug ID to track?
  2. Is a fix confirmed or planned for any JetPack 7.x release? (Not listed as fixed through R39.2 / JP7.2.)
  3. Any recommended workaround beyond clock-pinning that more fully avoids the trigger on R38.4 / JP7.1, given we can’t upgrade yet?

Happy to attach full dmesg, the complete hung-task stack traces, nvpmodel -q, jetson_clocks --show, and per-unit kernel build details. Let me know what’s most useful.

4

Hi,

Could you share the output of the command below when the GPU stall happens?

$ sudo dmesg

Thanks.

5

Sure:
dmesg.txt (106.8 KB)

6

Hi,

There is a known deadlock issue which is fixed in r39.2.

If the issue can still be reproduced with the patch, could you try it on the devkit?
Thanks.

7

The board vendor did no release yet a BSP supporting the new Jetpack / Linux release. Can you confirm it’s really the same issue? From the post it is not clear that the issue this patch solves had the same symptoms.

8

Hi,

The patch can be applied to the previous release directly.
You can find the details on the link shared above.

Thanks.