Summary
Our DeepStream 7.1 application hangs the whole AGX Orin under sustained heavy GPU load. CPU0 stops responding with no prior error, and the CUDA runtime’s cuda-EvtHandlr thread is the task running on it. Every interrupt serviced on CPU0 then times out (NVMe, HDA, host1x syncpoint waiters, BPMP i2c), and the board only recovers through a reset. No nvgpu or host1x fault is printed before the stall.
We’ve seen it three times on the same AGX Orin dev kit, and once on an Orin NX under a heavier configuration. The time to failure falls as load rises: ~18 minutes at our heaviest load, ~64 hours at a lighter one. The GPU was at 99–100 % each time. Temperature, power, memory and CPU were all normal up to the last sample.
We’d like to know whether this is a known issue in the 36.4.x GPU driver, whether a later release fixes it, and what extra debug output would help.
Environment
| Board | Jetson AGX Orin Developer Kit (64 GB) |
| JetPack / L4T | JetPack 6.2.1 (L4T R36.4.4, GCID 41062509), kernel 5.15.148-tegra |
| Power | nvpmodel MAXN, jetson_clocks on (GPU pinned at 1.3 GHz) |
| DeepStream | 7.1, in the DeepStream 7.1 L4T container (Docker) |
| Inference | nvinferserver (Triton), 2 TensorRT INT8 engines (PeopleNet and TrafficCamNet), batch 40, TensorRT 10.3 |
| Other GPU work | VPI background subtractor on the CUDA backend, in a custom GStreamer element. VPI 3.2.4 (libnvvpi3, host package from JetPack 6.2.1+b38) |
| CUDA | 12.6 (cuda-cudart-12-6 12.6.68, host package) |
| Storage | NVMe (Crucial P310 2 TB). SMART is clean: 0 media errors, 0 error-log entries. |
Pipeline
One process, one GStreamer pipeline
N × [appsrc (H.264 access units) → nvv4l2decoder → nvvidconv → capsfilter NV12/NVMM]
→ nvstreammux (new mux, batch-size 40, adaptive batching)
→ custom VPI motion element (CUDA backend, full-resolution frames, every 3rd frame)
→ nvinferserver PeopleNet (interval=3) → nvinferserver TrafficCamNet (interval=3)
→ nvdsanalytics → nvstreamdemux
→ N × [queue → nvvideoconvert → nvjpegenc → appsink] (≈1 fps per stream, on events only)
Inputs are 1080p or 1440p H.264 at 12 fps, simulated (one video file fanned out to N RTSP clients).
Loads at which it happened
| Load | Time to hang | GPU at ≥99 % (whole run) | NVDEC | Recovery |
|---|---|---|---|---|
| 25 × 1080p12, batch 20 | ~10 min | ~24 % | ~67 % | HW watchdog, ~8 min |
| 30 × 1080p12, batch 40 | ~64 h of uptime | ~23 % | ~78 % | ~44 min (kernel.panic=0`) |
| 8 × 1440p12, batch 40 | ~18 min | ~38 % | ~80 % (p95 90 %) | ~2.5 min (panic_on_rcu_stall=1) |
The GPU was at 99.9 % in the seconds before each hang. Before each hang the busiest CPU core was under 50 % and tj was 64–67 °C, with no thermal throttling. No OOM; RAM was under 50 % of 64 GB.
The same signature (cuda-EvtHandlr on CPU0 → RCU stall → watchdog reset) also showed up on an Orin NX running the pipeline with inference on every frame. There the log also had a host1x syncpoint dump with stuck waiters and, on one boot, nvgpu_tsg_unbind_channel ... err=-11.
Kernel log (from pstore dmesg-ramoops-0)
This is the clearest of the three, because kernel.panic_on_rcu_stall=1 was set:
[148754.297226] tegra-hda 3510000.hda: azx_get_response timeout, switching to polling mode: last cmd=0x004f0900
[148758.917109] ---- syncpts ----
... (all other syncpoints: 0 waiters)
[148758.917183] id 33 (33-ga10b_511_user) min 16273588 max 15387677 (1 waiters)
[148758.917185] id 34 (34-ga10b_510_user) min 16206321 max 15325502 (1 waiters)
[148769.796844] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[148769.796852] rcu: 0-...0: (1 GPs behind) idle=ed9/1/0x4000000000000002 softirq=23787676/23787677 fqs=1053
[148769.796859] (detected by 1, t=5252 jiffies, g=39082269, q=2954)
[148769.796862] Task dump for CPU 0:
[148769.796864] task:cuda-EvtHandlr state:R running task stack: 0 pid:1245241 ppid:1244965 flags:0x00000002
[148769.796870] Call trace:
[148769.796871] __switch_to+0x104/0x160
[148769.796883] 0x0
[148769.796890] Kernel panic - not syncing:
[148769.796891] RCU Stall
[148769.796894] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G O 5.15.148-tegra #1
[148769.796898] Hardware name: NVIDIA NVIDIA Jetson AGX Orin Developer Kit/Jetson, BIOS 36.4.4-gcid-41062509 06/16/2025
...
[148770.801915] SMP: stopping secondary CPUs
[148771.829690] SMP: failed to stop secondary CPUs 0-1,6,9
Kernel log (panic_on_rcu_stall=0)
CPU0 stalled with rcu: 0-...!: (0 ticks this GP), its softirq count never changed again, and there was no task dump at all. The other 11 cores ran for ~41 minutes, logging:
nvme nvme0: I/O ... timeout, completion polledon all queues (all NVMe MSIs are serviced on CPU0)- BPMP i2c, thermal-sensor and RTC reads failing with -62
- systemd service watchdog expiry for journald, logind and timesyncd
After that the board reset with no further message. One nvgpu channel timeout appeared in this log, owned by vulkaninfo, but its job was submitted ~30 s after the stall had already begun, so we think it got caught in the hang.
Kernel log
tegra-hda azx_get_response timeout → rcu_preempt detected stalls on CPUs/tasks: 0-... → Task dump for CPU 0: task:cuda-EvtHandlr state:R → NVMe completion polled timeouts → hardware watchdog reset. No host1x or nvgpu error was printed.
What we’ve ruled out
- NVMe/APST: the drive is healthy, and the NVMe timeouts start only after CPU0 has stalled (
completion polled). - Thermal and power: tj 64–67 °C, no throttle events, power steady at 45–47 W, no regulator or PMIC messages.
- Memory: no OOM, RAM under 50 %.
- Application faults: no CUDA, TensorRT or GStreamer errors were logged before the hang. Our own inference-stall watchdog didn’t trigger, and frames kept flowing until CPU0 stopped.
Questions
- Is a CPU hard hang inside the GPU driver, under sustained 100 % GPU load from mixed CUDA (VPI) and TensorRT/Triton work, a known issue on R36.4.x? Is it fixed in a later L4T release?
- How can we capture the stack of the stuck CPU? Is there a supported way on Orin to get a backtrace of a CPU stuck with interrupts masked (pseudo-NMI,
sysrq-l, or a kernel config option)? - Are there nvgpu or host1x debug options (debugfs, ftrace events, kernel cmdline) you’d like us to turn on for the next repro? We can reproduce within ~20 minutes at the 18 × 1440p load.