GPU hang

Continuing from the previous ticket: GPU not responding since the issue is still present.

To recap, our workload is video decode and inferencing with CUDA, and we collect periodic GPU health metrics from the device by looking at tegrastats. After running for a long time, we observed that our containers running decoding/inferencing would fail the periodic health checks. And any further calls to tegrastats would also get stuck. We do not have exact steps to reproduce, but issue seems to arise when there is intense GPU load.

Observations:

  1. nvidia-container-runtime process instances getting stuck in uninterruptible sleep (D) state. From the stack trace, it seems to be stuck in some iotctl calls
    # ps -eo stat,pid,comm | grep D | tail -n3
    D 4173139 nvidia-containe
    D 4178598 nvidia-containe
    D 4190689 nvidia-containe
    # cat /proc/4173139/stack
    [<0>] __switch_to+0xfc/0x150
    [<0>] rpm_resume+0x190/0x760
    [<0>] __pm_runtime_resume+0x4c/0xb0
    [<0>] gk20a_busy+0x130/0x1f0 [nvgpu]
    [<0>] gk20a_ctrl_ioctl_gpu_characteristics+0x7c/0x770 [nvgpu]
    [<0>] gk20a_ctrl_dev_ioctl+0x1644/0x2700 [nvgpu]
    [<0>] __arm64_sys_ioctl+0xb8/0x100
    [<0>] invoke_syscall+0x90/0x120
    [<0>] el0_svc_common.constprop.0+0xf0/0x110
    [<0>] do_el0_svc+0x3c/0xa0
    [<0>] el0_svc+0x20/0x60
    [<0>] el0t_64_sync_handler+0x118/0x130
    [<0>] el0t_64_sync+0x1a4/0x1a
    // stuck at syscall 29 ioctl
    # cat /proc/4173139/syscall
    29 0x8 0xc0104705 0xffffc6760020 0x10 0xffffc6760178 0xaaaaeeda7130 0xffffc675ff70 0xffffb4dfdf50

  2. tegrastats getting stuck

    # ps -eo stat,pid,comm | grep tegrastats
    D 390214 tegrastats
    # cat /proc/390214/stack
    [<0>] __switch_to+0xfc/0x150
    [<0>] rpm_resume+0x190/0x760
    [<0>] __pm_runtime_resume+0x4c/0xb0
    [<0>] gk20a_busy+0x130/0x1f0 [nvgpu]
    [<0>] gk20a_load_show+0xa4/0xd0 [nvgpu]
    [<0>] dev_attr_show+0x44/0x8c
    [<0>] sysfs_kf_seq_show+0x98/0x150
    [<0>] kernfs_seq_show+0x4c/0x60
    [<0>] seq_read_iter+0xec/0x4a0
    [<0>] kernfs_fop_read_iter+0x198/0x1e0
    [<0>] new_sync_read+0x114/0x1c4
    [<0>] vfs_read+0x110/0x1c0
    [<0>] ksys_read+0x88/0x11c
    [<0>] __arm64_sys_read+0x2c/0x40
    [<0>] invoke_syscall+0x90/0x120
    [<0>] el0_svc_common.constprop.0+0xf0/0x110
    [<0>] do_el0_svc+0x3c/0xa0
    [<0>] el0_svc+0x20/0x60
    [<0>] el0t_64_sync_handler+0x118/0x130
    [<0>] el0t_64_sync+0x1a4/0x1a8
    // stuck at syscall 63 read
    # cat /proc/390214/syscall
    # 63 0x3 0xaaab09aaffe0 0x1000 0x0 0xf 0x1 0xffffcd75f960 0xffff84118d18
    // cannot be killed with kill -9
    # kill -9 390214
    # ps aux | grep tegrastats390214 root 0:00 tegrastats
    414701 root 0:00 grep tegrastats

  3. GPU runtime status returns “suspending”

    # cat /sys/devices/platform/gpu.0/power/runtime_status
    suspending

Any clues on what could be causing the GPU hang?

Attaching all the logs we could get here, if that helps

agx-orin-ps.log (469.4 KB)

agx-orin-nvidia-bug-report.log (186.7 KB)

The processes are stuck in D state during the rpm_resume() call, following this path: gk20a_busy() → pm_runtime_get_sync() → __pm_runtime_resume() → rpm_resume()

The issue occurs because:

  1. rpm_resume() puts the task in uninterruptible state while waiting for device runtime suspend to complete (linux-jammy-nvidia-tegra/drivers/base/power/runtime.c at 70b1f13ff48308ab0d1b4b4cbc6109fddd5da7d4 · OE4T/linux-jammy-nvidia-tegra · GitHub)

  2. The device should transition to SUSPENDED state, but it’s stuck in SUSPENDING: linux-jammy-nvidia-tegra/drivers/base/power/runtime.c at 70b1f13ff48308ab0d1b4b4cbc6109fddd5da7d4 · OE4T/linux-jammy-nvidia-tegra · GitHub

  3. Since the status never changes to SUSPENDED, rpm_resume() is never woken up

  4. This suggests the device runtime suspend callback function, gk20a_pm_runtime_suspend() , is hanging.

Linux kernel that we are using: GitHub - OE4T/linux-jammy-nvidia-tegra at oe4t-patches-l4t-r36.4-1012.12

nvgpu driver that we are using: GitHub - OE4T/linux-nvgpu at l4t/l4t-r36.4

Any idea why gk20a_pm_runtime_suspend() would hang?

Hi,
Please try the stress test(CPU + GPU) on your setup and see if the issue is present:
Making sure you're not a bot!

If the issue is present when running the test, we can set up the same on developer kit to replicate it and check.

Hi, we will try this test.