Continuing from the previous ticket: GPU not responding since the issue is still present.
- Device: Jetson AGX Orin 64GB
- Jetpack version: 6.1
- Kernel patches applied:
To recap, our workload is video decode and inferencing with CUDA, and we collect periodic GPU health metrics from the device by looking at tegrastats. After running for a long time, we observed that our containers running decoding/inferencing would fail the periodic health checks. And any further calls to tegrastats would also get stuck. We do not have exact steps to reproduce, but issue seems to arise when there is intense GPU load.
Observations:
-
nvidia-container-runtimeprocess instances getting stuck in uninterruptible sleep (D) state. From the stack trace, it seems to be stuck in someiotctlcalls
# ps -eo stat,pid,comm | grep D | tail -n3
D 4173139 nvidia-containe
D 4178598 nvidia-containe
D 4190689 nvidia-containe
# cat /proc/4173139/stack
[<0>] __switch_to+0xfc/0x150
[<0>] rpm_resume+0x190/0x760
[<0>] __pm_runtime_resume+0x4c/0xb0
[<0>] gk20a_busy+0x130/0x1f0 [nvgpu]
[<0>] gk20a_ctrl_ioctl_gpu_characteristics+0x7c/0x770 [nvgpu]
[<0>] gk20a_ctrl_dev_ioctl+0x1644/0x2700 [nvgpu]
[<0>] __arm64_sys_ioctl+0xb8/0x100
[<0>] invoke_syscall+0x90/0x120
[<0>] el0_svc_common.constprop.0+0xf0/0x110
[<0>] do_el0_svc+0x3c/0xa0
[<0>] el0_svc+0x20/0x60
[<0>] el0t_64_sync_handler+0x118/0x130
[<0>] el0t_64_sync+0x1a4/0x1a
// stuck at syscall 29 ioctl
# cat /proc/4173139/syscall
29 0x8 0xc0104705 0xffffc6760020 0x10 0xffffc6760178 0xaaaaeeda7130 0xffffc675ff70 0xffffb4dfdf50 -
tegrastatsgetting stuck# ps -eo stat,pid,comm | grep tegrastats
D 390214 tegrastats
# cat /proc/390214/stack
[<0>] __switch_to+0xfc/0x150
[<0>] rpm_resume+0x190/0x760
[<0>] __pm_runtime_resume+0x4c/0xb0
[<0>] gk20a_busy+0x130/0x1f0 [nvgpu]
[<0>] gk20a_load_show+0xa4/0xd0 [nvgpu]
[<0>] dev_attr_show+0x44/0x8c
[<0>] sysfs_kf_seq_show+0x98/0x150
[<0>] kernfs_seq_show+0x4c/0x60
[<0>] seq_read_iter+0xec/0x4a0
[<0>] kernfs_fop_read_iter+0x198/0x1e0
[<0>] new_sync_read+0x114/0x1c4
[<0>] vfs_read+0x110/0x1c0
[<0>] ksys_read+0x88/0x11c
[<0>] __arm64_sys_read+0x2c/0x40
[<0>] invoke_syscall+0x90/0x120
[<0>] el0_svc_common.constprop.0+0xf0/0x110
[<0>] do_el0_svc+0x3c/0xa0
[<0>] el0_svc+0x20/0x60
[<0>] el0t_64_sync_handler+0x118/0x130
[<0>] el0t_64_sync+0x1a4/0x1a8
// stuck at syscall 63 read
# cat /proc/390214/syscall
# 63 0x3 0xaaab09aaffe0 0x1000 0x0 0xf 0x1 0xffffcd75f960 0xffff84118d18
// cannot be killed with kill -9
# kill -9 390214
# ps aux | grep tegrastats390214 root 0:00 tegrastats
414701 root 0:00 grep tegrastats -
GPU runtime status returns “suspending”
# cat /sys/devices/platform/gpu.0/power/runtime_status
suspending
Any clues on what could be causing the GPU hang?
Attaching all the logs we could get here, if that helps
agx-orin-ps.log (469.4 KB)
agx-orin-nvidia-bug-report.log (186.7 KB)