Orin Nano program crashes and device crashes during restart stability testing

Robotics & Edge Computing Jetson Systems Jetson Orin Nano
,
1

Hi:
My HardwareDevice: orin nano 4G
software version:JetPack 5.1.4

I added a service on Orin Nano, and after 140 seconds of starting the service, the device executed the reboot - f command to perform a soft restart. In long-term stability testing, after 802 soft restarts, Orin Nano’s Ubuntu interface froze and could not be operated, with no response from the keyboard and mouse. When using the serial port to query dmesg information, the crash message is as follows:

[   18.298317] nvidia-modeset: Loading NVIDIA UNIX Open Kernel Mode Setting Driver for aarch64  35.5.0  Release Build  (buildbrain@mobile-u64-6429-d8000)  Wed Aug 28 01:59:43 PDT 2024
[   18.897508] NVRM nvAssertFailedNoLog: Assertion failed: minRequiredIsoBandwidthKBPS <= clientBwValues[DISPLAY_ICC_BW_CLIENT_EXT].minRequiredIsoBandwidthKBPS @ kern_disp_0402.c:111
[   18.897522] CPU: 3 PID: 144 Comm: kworker/u12:2 Tainted: G           OE     5.10.216-tegra #2
[   18.897525] Hardware name: NVIDIA NVIDIA Orin Nano Developer Kit/Jetson, BIOS 6.0-37391689 08/28/2024
[   18.897537] Workqueue: dce-async-ipc-wq dce_client_async_event_work
[   18.897541] Call trace:
[   18.897551]  dump_backtrace+0x0/0x1e0
[   18.897553]  show_stack+0x30/0x40
[   18.897559]  dump_stack+0xf0/0x130
[   18.897623]  os_dump_stack+0x18/0x20 [nvidia]
[   18.897678]  tlsEntryGet+0x130/0x138 [nvidia]
[   18.897728]  kdispArbAndAllocDisplayBandwidth_v04_02+0x274/0x288 [nvidia]
[   18.897777]  kdispInvokeDisplayModesetCallback_KERNEL+0xa8/0xf0 [nvidia]
[   18.897826]  dceclientHandleAsyncRpcCallback+0x17c/0x2f0 [nvidia]
[   18.897874]  initVGXSpecificRegistry+0x1dc/0x300 [nvidia]
[   18.897879]  dce_client_async_event_work+0x8c/0x190
[   18.897883]  process_one_work+0x1c4/0x4c0
[   18.897886]  worker_thread+0x54/0x450
[   18.897888]  kthread+0x148/0x170
[   18.897891]  ret_from_fork+0x10/0x18
[   20.087718] fuse: init (API version 7.32)
[   21.690968] ACK 04 d4
[   21.692326] ACK 04 d4
[   21.722039] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
[   22.521635] rfkill: input handler disabled
[  157.861079] kvm: exiting hardware virtualization
[  158.013835] nvgpu: 17000000.ga10b     nvgpu_submit_nondeterministic:678  [ERR]  failed to host gk20a to submit gpfifo
[  158.024882] nvgpu: 17000000.ga10b     nvgpu_submit_nondeterministic:680  [ERR]  gnome-shell
[  158.033652] nvgpu: 17000000.ga10b     nvgpu_submit_nondeterministic:678  [ERR]  failed to host gk20a to submit gpfifo
[  158.044626] nvgpu: 17000000.ga10b     nvgpu_submit_nondeterministic:680  [ERR]  gnome-shell
[  158.053468] nvgpu: 17000000.ga10b     nvgpu_submit_nondeterministic:678  [ERR]  failed to host gk20a to submit gpfifo
[  158.064428] nvgpu: 17000000.ga10b     nvgpu_submit_nondeterministic:680  [ERR]  gnome-shell
[  158.073292] nvgpu: 17000000.ga10b     nvgpu_submit_nondeterministic:678  [ERR]  failed to host gk20a to submit gpfifo
[  158.076814] nvgpu: 17000000.ga10b     nvgpu_submit_nondeterministic:678  [ERR]  failed to host gk20a to submit gpfifo
[  158.084252] nvgpu: 17000000.ga10b     nvgpu_submit_nondeterministic:680  [ERR]  gnome-shell
[  158.084272] nvgpu: 17000000.ga10b     nvgpu_submit_nondeterministic:678  [ERR]  failed to host gk20a to submit gpfifo
[  158.095223] nvgpu: 17000000.ga10b     nvgpu_submit_nondeterministic:680  [ERR]  Xorg
[  158.103797] nvgpu: 17000000.ga10b     nvgpu_submit_nondeterministic:680  [ERR]  gnome-shell
[  158.103808] nvgpu: 17000000.ga10b         nvgpu_ioctl_tsg_dev_ioctl:850  [ERR]  failed to host gk20a for ioctl cmd: 0x40045402
[  158.103818] nvgpu: 17000000.ga10b             gk20a_channel_release:459  [ERR]  failed to release a channel!
[  158.207300] nvgpu: 17000000.ga10b     nvgpu_submit_nondeterministic:678  [ERR]  failed to host gk20a to submit gpfifo
[  158.218298] nvgpu: 17000000.ga10b     nvgpu_submit_nondeterministic:680  [ERR]  Xorg
[  158.226434] nvgpu: 17000000.ga10b     nvgpu_submit_nondeterministic:678  [ERR]  failed to host gk20a to submit gpfifo
[  158.237381] nvgpu: 17000000.ga10b     nvgpu_submit_nondeterministic:680  [ERR]  Xorg
[  158.245406] nvgpu: 17000000.ga10b     nvgpu_submit_nondeterministic:678  [ERR]  failed to host gk20a to submit gpfifo
[  158.256342] nvgpu: 17000000.ga10b     nvgpu_submit_nondeterministic:680  [ERR]  Xorg
[  158.264361] nvgpu: 17000000.ga10b     nvgpu_submit_nondeterministic:678  [ERR]  failed to host gk20a to submit gpfifo
[  158.275299] nvgpu: 17000000.ga10b     nvgpu_submit_nondeterministic:680  [ERR]  Xorg
[  158.283323] nvgpu: 17000000.ga10b     nvgpu_submit_nondeterministic:678  [ERR]  failed to host gk20a to submit gpfifo
[  158.294253] nvgpu: 17000000.ga10b     nvgpu_submit_nondeterministic:680  [ERR]  Xorg
[  158.302277] nvgpu: 17000000.ga10b     nvgpu_submit_nondeterministic:678  [ERR]  failed to host gk20a to submit gpfifo
[  158.313207] nvgpu: 17000000.ga10b     nvgpu_submit_nondeterministic:680  [ERR]  Xorg
[  158.321227] nvgpu: 17000000.ga10b     nvgpu_submit_nondeterministic:678  [ERR]  failed to host gk20a to submit gpfifo
[  158.332167] nvgpu: 17000000.ga10b     nvgpu_submit_nondeterministic:680  [ERR]  Xorg
[  158.340198] nvgpu: 17000000.ga10b     nvgpu_submit_nondeterministic:678  [ERR]  failed to host gk20a to submit gpfifo
[  158.351132] nvgpu: 17000000.ga10b     nvgpu_submit_nondeterministic:680  [ERR]  Xorg
[  158.359153] nvgpu: 17000000.ga10b     nvgpu_submit_nondeterministic:678  [ERR]  failed to host gk20a to submit gpfifo
[  158.370084] nvgpu: 17000000.ga10b     nvgpu_submit_nondeterministic:680  [ERR]  Xorg
[  158.602990] nvgpu: 17000000.ga10b gk20a_ctrl_ioctl_gpu_characteristics:370  [ERR]  failed to power on gpu
[  158.615089] nvgpu: 17000000.ga10b gk20a_ctrl_ioctl_gpu_characteristics:370  [ERR]  failed to power on gpu
[  158.675845] nvgpu: 17000000.ga10b gk20a_ctrl_ioctl_gpu_characteristics:370  [ERR]  failed to power on gpu
[  158.736606] nvgpu: 17000000.ga10b gk20a_ctrl_ioctl_gpu_characteristics:370  [ERR]  failed to power on gpu
[  158.746696] nvgpu: 17000000.ga10b gk20a_ctrl_ioctl_gpu_characteristics:370  [ERR]  failed to power on gpu
[  158.757406] nvgpu: 17000000.ga10b gk20a_ctrl_ioctl_gpu_characteristics:370  [ERR]  failed to power on gpu
[  158.816666] nvgpu: 17000000.ga10b gk20a_ctrl_ioctl_gpu_characteristics:370  [ERR]  failed to power on gpu
[  158.843161] nvgpu: 17000000.ga10b     nvgpu_submit_nondeterministic:678  [ERR]  failed to host gk20a to submit gpfifo
[  158.854147] nvgpu: 17000000.ga10b     nvgpu_submit_nondeterministic:680  [ERR]  Xorg
[  158.862251] nvgpu: 17000000.ga10b     nvgpu_submit_nondeterministic:678  [ERR]  failed to host gk20a to submit gpfifo
[  158.873213] nvgpu: 17000000.ga10b     nvgpu_submit_nondeterministic:680  [ERR]  Xorg
[  159.027872] nvgpu: 17000000.ga10b gk20a_ctrl_ioctl_gpu_characteristics:370  [ERR]  failed to power on gpu
[  159.039595] nvgpu: 17000000.ga10b gk20a_ctrl_ioctl_gpu_characteristics:370  [ERR]  failed to power on gpu
[  159.100878] nvgpu: 17000000.ga10b gk20a_ctrl_ioctl_gpu_characteristics:370  [ERR]  failed to power on gpu
[  159.162345] nvgpu: 17000000.ga10b gk20a_ctrl_ioctl_gpu_characteristics:370  [ERR]  failed to power on gpu
[  159.172447] nvgpu: 17000000.ga10b gk20a_ctrl_ioctl_gpu_characteristics:370  [ERR]  failed to power on gpu

Restore normal operation after device restart.
Complete log:
orin_nano_crash.log (87.4 KB)

3

Is this issue able to reproduce with NV devkit?

4

hi, WayneWWW:
I have not tested this issue on the devKit, and currently I do not have any avaliable devKit for testing.

5

please try to get a NV devkit and try to see if you could reproduce it. Otherwise we cannot help.

6

Hi, WayneWWW:
The devkit we purchased experienced a short circuit that burned out one Orin Nano core board, rendering it currently unusable. If this phenomenon occurs, could we mitigate the issue by forcing a device reboot through the watchdog timer?

7

Hi, WayneWWW:
I searched for the same error print in the forum and saw feedback that it is a known issue：https://forums.developer.nvidia.com/t/jetson-orin-nano-system-stuck-when-shutdown/263161. I would like to confirm if this problem has been fixed in 5.1.4 or if some patches need to be added?

8

No, this issue is not a known issue. Kay gave out an answer which is not precise.

9

From the print logs, it seems to be related to GPU, but it is not related to the motherboard used. Our motherboard has only expanded one network port without any other changes. Do you have any suggestions for troubleshooting this issue?

10

No, I can only suggest you to get a NV devkit and try to reproduce. The is the how the debug process is doing here.