AGX Orin (JetPack 6.2.1): CPU0 hard hang with `cuda-EvtHandlr`, under sustained GPU load (DeepStream 7.1 + Triton + VPI)

Summary

Our DeepStream 7.1 application hangs the whole AGX Orin under sustained heavy GPU load. CPU0 stops responding with no prior error, and the CUDA runtime’s cuda-EvtHandlr thread is the task running on it. Every interrupt serviced on CPU0 then times out (NVMe, HDA, host1x syncpoint waiters, BPMP i2c), and the board only recovers through a reset. No nvgpu or host1x fault is printed before the stall.

We’ve seen it three times on the same AGX Orin dev kit, and once on an Orin NX under a heavier configuration. The time to failure falls as load rises: ~18 minutes at our heaviest load, ~64 hours at a lighter one. The GPU was at 99–100 % each time. Temperature, power, memory and CPU were all normal up to the last sample.

We’d like to know whether this is a known issue in the 36.4.x GPU driver, whether a later release fixes it, and what extra debug output would help.

Environment

Board Jetson AGX Orin Developer Kit (64 GB)
JetPack / L4T JetPack 6.2.1 (L4T R36.4.4, GCID 41062509), kernel 5.15.148-tegra
Power nvpmodel MAXN, jetson_clocks on (GPU pinned at 1.3 GHz)
DeepStream 7.1, in the DeepStream 7.1 L4T container (Docker)
Inference nvinferserver (Triton), 2 TensorRT INT8 engines (PeopleNet and TrafficCamNet), batch 40, TensorRT 10.3
Other GPU work VPI background subtractor on the CUDA backend, in a custom GStreamer element. VPI 3.2.4 (libnvvpi3, host package from JetPack 6.2.1+b38)
CUDA 12.6 (cuda-cudart-12-6 12.6.68, host package)
Storage NVMe (Crucial P310 2 TB). SMART is clean: 0 media errors, 0 error-log entries.

Pipeline

One process, one GStreamer pipeline

N × [appsrc (H.264 access units) → nvv4l2decoder → nvvidconv → capsfilter NV12/NVMM]
  → nvstreammux (new mux, batch-size 40, adaptive batching)
  → custom VPI motion element (CUDA backend, full-resolution frames, every 3rd frame)
  → nvinferserver PeopleNet (interval=3) → nvinferserver TrafficCamNet (interval=3)
  → nvdsanalytics → nvstreamdemux
  → N × [queue → nvvideoconvert → nvjpegenc → appsink]   (≈1 fps per stream, on events only)

Inputs are 1080p or 1440p H.264 at 12 fps, simulated (one video file fanned out to N RTSP clients).

Loads at which it happened

Load Time to hang GPU at ≥99 % (whole run) NVDEC Recovery
25 × 1080p12, batch 20 ~10 min ~24 % ~67 % HW watchdog, ~8 min
30 × 1080p12, batch 40 ~64 h of uptime ~23 % ~78 % ~44 min (kernel.panic=0`)
8 × 1440p12, batch 40 ~18 min ~38 % ~80 % (p95 90 %) ~2.5 min (panic_on_rcu_stall=1)

The GPU was at 99.9 % in the seconds before each hang. Before each hang the busiest CPU core was under 50 % and tj was 64–67 °C, with no thermal throttling. No OOM; RAM was under 50 % of 64 GB.

The same signature (cuda-EvtHandlr on CPU0 → RCU stall → watchdog reset) also showed up on an Orin NX running the pipeline with inference on every frame. There the log also had a host1x syncpoint dump with stuck waiters and, on one boot, nvgpu_tsg_unbind_channel ... err=-11.

Kernel log (from pstore dmesg-ramoops-0)

This is the clearest of the three, because kernel.panic_on_rcu_stall=1 was set:

[148754.297226] tegra-hda 3510000.hda: azx_get_response timeout, switching to polling mode: last cmd=0x004f0900
[148758.917109] ---- syncpts ----
...                (all other syncpoints: 0 waiters)
[148758.917183] id 33 (33-ga10b_511_user) min 16273588 max 15387677 (1 waiters)
[148758.917185] id 34 (34-ga10b_510_user) min 16206321 max 15325502 (1 waiters)
[148769.796844] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[148769.796852] rcu:         0-...0: (1 GPs behind) idle=ed9/1/0x4000000000000002 softirq=23787676/23787677 fqs=1053
[148769.796859]      (detected by 1, t=5252 jiffies, g=39082269, q=2954)
[148769.796862] Task dump for CPU 0:
[148769.796864] task:cuda-EvtHandlr  state:R  running task     stack:    0 pid:1245241 ppid:1244965 flags:0x00000002
[148769.796870] Call trace:
[148769.796871]  __switch_to+0x104/0x160
[148769.796883]  0x0
[148769.796890] Kernel panic - not syncing:
[148769.796891] RCU Stall
[148769.796894] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G           O      5.15.148-tegra #1
[148769.796898] Hardware name: NVIDIA NVIDIA Jetson AGX Orin Developer Kit/Jetson, BIOS 36.4.4-gcid-41062509 06/16/2025
...
[148770.801915] SMP: stopping secondary CPUs
[148771.829690] SMP: failed to stop secondary CPUs 0-1,6,9

Kernel log (panic_on_rcu_stall=0)

CPU0 stalled with rcu: 0-...!: (0 ticks this GP), its softirq count never changed again, and there was no task dump at all. The other 11 cores ran for ~41 minutes, logging:

  • nvme nvme0: I/O ... timeout, completion polled on all queues (all NVMe MSIs are serviced on CPU0)
  • BPMP i2c, thermal-sensor and RTC reads failing with -62
  • systemd service watchdog expiry for journald, logind and timesyncd

After that the board reset with no further message. One nvgpu channel timeout appeared in this log, owned by vulkaninfo, but its job was submitted ~30 s after the stall had already begun, so we think it got caught in the hang.

Kernel log

tegra-hda azx_get_response timeout → rcu_preempt detected stalls on CPUs/tasks: 0-... → Task dump for CPU 0: task:cuda-EvtHandlr state:R → NVMe completion polled timeouts → hardware watchdog reset. No host1x or nvgpu error was printed.

What we’ve ruled out

  • NVMe/APST: the drive is healthy, and the NVMe timeouts start only after CPU0 has stalled (completion polled).
  • Thermal and power: tj 64–67 °C, no throttle events, power steady at 45–47 W, no regulator or PMIC messages.
  • Memory: no OOM, RAM under 50 %.
  • Application faults: no CUDA, TensorRT or GStreamer errors were logged before the hang. Our own inference-stall watchdog didn’t trigger, and frames kept flowing until CPU0 stopped.

Questions

  1. Is a CPU hard hang inside the GPU driver, under sustained 100 % GPU load from mixed CUDA (VPI) and TensorRT/Triton work, a known issue on R36.4.x? Is it fixed in a later L4T release?
  2. How can we capture the stack of the stuck CPU? Is there a supported way on Orin to get a backtrace of a CPU stuck with interrupts masked (pseudo-NMI, sysrq-l, or a kernel config option)?
  3. Are there nvgpu or host1x debug options (debugfs, ftrace events, kernel cmdline) you’d like us to turn on for the next repro? We can reproduce within ~20 minutes at the 18 × 1440p load.

Thanks for the detailed report. Attached is a small repro app that mirrors your pipeline:

repro_pipeline.cpp (24.1 KB)

rtspsrc (TCP) -> depay -> h264parse -> appsink => appsrc -> nvv4l2decoder -> nvvideoconvert  ->
new nvstreammux -> VPI BackgroundSubtractor (every 3rd frame, full-res) + 2x nvinferserver
(TrafficCamNet/PeopleNet INT8, interval 3) -> JPEG ~1 fps/stream.

I ran it on AGX Orin, JP-7.1 + DS-9.1 + MAXN + 12 x 1440p @ 12 fps for 77 min, no hang, no RCU stall, no
nvgpu/host1x errors in dmesg.

Usage: GPU avg ~58% (>=99% in ~24% of samples), CPU avg ~11% (busiest core ~19%), tj max ~64 C, RAM ~16.8 GB flat.

So we could not reproduce it on the current stack. We usually don’t backport to older software stacks, so please try JetPack 7.1 / DS 9.1.

For your question:

This issue has not been reproduced internally; you may want to try a new software stack.

You can examine the kernel logs after a reboot, but you will first need to enable the relevant kernel configuration options—information on how to do this can be found in the Jetson community.

cd /sys/fs/pstore/

  • ftrace: enable the gk20a:* and host1x:* events, dump on panic (ftrace_dump_on_oops=1).
  • debugfs /sys/kernel/debug/gpu.0: record railgate_* and timeouts_enabled, ch_wdt_init_limit_ms,
    enable trace_cmdbuf.
    (Names checked on R39.2; they may differ on R36.4.4.)