Experienced a crash of a Jetson TX2 during a stress test in a Temp chamber after approx 30 min at 55c.
I was testing a custom cooling solution with fans powered externally from tegra.
In order to stress the system i was running several instances of both Cuda simulation demos and the Vision Works video stabilization demo.
Trying to find the exact cause of the crash, while over temperature seems like the obvious reason it does not appear to be the case as temps are below throttling temperature and cpu clocks remain unrestricted up until the crash.
Any help or insight would be appreciated.
Note: the system time and date are incorrect, system was offline for quite some time before this test
L4T version
# R32 (release), REVISION: 1.0, GCID: 14531094, BOARD: t186ref, EABI: aarch64, DATE: Wed Mar 13 07:41:08 UTC 2019
tegrastats output at time of crash
[2019-05-23 20:30:45] RAM 4776/7852MB (lfb 238x4MB) CPU [36%@1574,53%@2035,60%@2035,27%@1574,26%@1574,29%@1574] EMC_FREQ 0% GR3D_FREQ 59% PLL@74C MCPU@74C PMIC@100C Tboard@67C GPU@72.5C BCPU@74C thermal@73.4C Tdiode@72.75C VDD_SYS_GPU 2218/2397 VDD_SYS_SOC 2064/1979 VDD_4V0_WIFI 0/10 VDD_IN 9529/9985 VDD_SYS_CPU 2524/2668 VDD_SYS_DDR 2040/2059
[2019-05-23 20:30:46] RAM 4776/7852MB (lfb 238x4MB) CPU [38%@2035,62%@2035,52%@2035,34%@2035,28%@2035,25%@2035] EMC_FREQ 0% GR3D_FREQ 72% PLL@74C MCPU@74C PMIC@100C Tboard@67C GPU@72.5C BCPU@74C thermal@73.4C Tdiode@73C VDD_SYS_GPU 2218/2397 VDD_SYS_SOC 2064/1979 VDD_4V0_WIFI 0/10 VDD_IN 9682/9985 VDD_SYS_CPU 2601/2668 VDD_SYS_DDR 2040/2059
[2019-05-23 20:30:48] RAM 4777/7852MB (lfb 238x4MB) CPU [37%@2035,60%@2035,58%@2035,30%@2035,29%@2035,29%@2035] EMC_FREQ 0% GR3D_FREQ 47% PLL@74C MCPU@74C PMIC@100C Tboard@67C GPU@72.5C BCPU@74C thermal@73.4C Tdiode@73C VDD_SYS_GPU 2447/2397 VDD_SYS_SOC 2064/1979 VDD_4V0_WIFI 0/10 VDD_IN 9950/9985 VDD_SYS_CPU 2601/2668 VDD_SYS_DDR 2040/2059
[2019-05-23 20:30:49] RAM 4777/7852MB (lfb 238x4MB) CPU [39%@1881,56%@2035,61%@2035,28%@1881,30%@1881,35%@1881] EMC_FREQ 0% GR3D_FREQ 71% PLL@74C MCPU@74C PMIC@100C Tboard@67C GPU@72.5C BCPU@74C thermal@73.4C Tdiode@73.25C VDD_SYS_GPU 2523/2397 VDD_SYS_SOC 2064/1979 VDD_4V0_WIFI 0/10 VDD_IN 10180/9985 VDD_SYS_CPU 2677/2668 VDD_SYS_DDR 2097/2059
[2019-05-23 20:30:50] RAM 4777/7852MB (lfb 238x4MB) CPU [39%@2035,57%@2035,58%@2035,28%@2035,29%@2035,28%@2035] EMC_FREQ 0% GR3D_FREQ 66% PLL@74C MCPU@74C PMIC@100C Tboard@67C GPU@72.5C BCPU@74C thermal@73.4C Tdiode@73.25C VDD_SYS_GPU 2753/2397 VDD_SYS_SOC 2064/1979 VDD_4V0_WIFI 0/10 VDD_IN 10295/9985 VDD_SYS_CPU 2601/2668 VDD_SYS_DDR 2059/2059
[2019-05-23 20:30:51] RAM 4777/7852MB (lfb 238x4MB) CPU [37%@2035,60%@2035,53%@2035,24%@2035,30%@2035,35%@2035] EMC_FREQ 0% GR3D_FREQ 77% PLL@74C MCPU@74C PMIC@100C Tboard@67C GPU@72.5C BCPU@74C thermal@73.7C Tdiode@73C VDD_SYS_GPU 2447/2397 VDD_SYS_SOC 2064/1979 VDD_4V0_WIFI 0/10 VDD_IN 9988/9985 VDD_SYS_CPU 2677/2668 VDD_SYS_DDR 2097/2059
[2019-05-23 20:30:52] RAM 4777/7852MB (lfb 238x4MB) CPU [27%@2035,60%@2035,56%@2035,39%@2035,26%@2035,32%@2035] EMC_FREQ 0% GR3D_FREQ 81% PLL@74C MCPU@74C PMIC@100C Tboard@67C GPU@72.5C BCPU@74C thermal@73.4C Tdiode@73C VDD_SYS_GPU 2370/2397 VDD_SYS_SOC 2064/1979 VDD_4V0_WIFI 0/10 VDD_IN 9988/9985 VDD_SYS_CPU 2601/2668 VDD_SYS_DDR 2059/2059
[2019-05-23 20:30:53] RAM 4777/7852MB (lfb 238x4MB) CPU [27%@2035,59%@2035,56%@2035,40%@2035,30%@2035,27%@2035] EMC_FREQ 0% GR3D_FREQ 75% PLL@74C MCPU@74C PMIC@100C Tboard@67C GPU@72.5C BCPU@74C thermal@73.4C Tdiode@73C VDD_SYS_GPU 2218/2397 VDD_SYS_SOC 1988/1979 VDD_4V0_WIFI 0/10 VDD_IN 9682/9985 VDD_SYS_CPU 2524/2668 VDD_SYS_DDR 2040/2059
[2019-05-23 20:30:54] RAM 4777/7852MB (lfb 238x4MB) CPU [32%@2035,55%@2035,56%@2035,34%@2035,34%@2035,34%@2035] EMC_FREQ 0% GR3D_FREQ 77% PLL@74C MCPU@74C PMIC@100C Tboard@67C GPU@72.5C BCPU@74C thermal@73.4C Tdiode@72.75C VDD_SYS_GPU 2294/2397 VDD_SYS_SOC 1988/1979 VDD_4V0_WIFI 0/10 VDD_IN 9721/9985 VDD_SYS_CPU 2677/2668 VDD_SYS_DDR 2059/2059
[2019-05-23 20:30:55] RAM 4778/7852MB (lfb 238x4MB) CPU [31%@2035,59%@2035,56%@2035,39%@2035,29%@2035,30%@2035] EMC_FREQ 0% GR3D_FREQ 77% PLL@74C MCPU@74C PMIC@100C Tboard@67C GPU@72.5C BCPU@74C thermal@73.4C Tdiode@73.25C VDD_SYS_GPU 2523/2397 VDD_SYS_SOC 1988/1979 VDD_4V0_WIFI 0/10 VDD_IN 10027/9985 VDD_SYS_CPU 2601/2668 VDD_SYS_DDR 2097/2059
[2019-05-23 20:30:56] RAM 4778/7852MB (lfb 238x4MB) CPU [37%@1881,54%@2035,53%@2035,34%@1881,31%@1881,33%@1881] EMC_FREQ 0% GR3D_FREQ 55% PLL@74C MCPU@74C PMIC@100C Tboard@67C GPU@72.5C BCPU@74C thermal@73.4C Tdiode@73C VDD_SYS_GPU 2370/2397 VDD_SYS_SOC 1988/1979 VDD_4V0_WIFI 0/10 VDD_IN 9988/9985 VDD_SYS_CPU 2677/2668 VDD_SYS_DDR 2040/2059
[2019-05-23 20:30:57] RAM 4730/7852MB (lfb 238x4MB) CPU [45%@1113,63%@2035,65%@2035,45%@1113,25%@1113,45%@1113] EMC_FREQ 0% GR3D_FREQ 0% PLL@73.5C MCPU@73.5C PMIC@100C Tboard@67C GPU@71C BCPU@73.5C thermal@73.4C Tdiode@71.75C VDD_SYS_GPU 1224/2397 VDD_SYS_SOC 1912/1979 VDD_4V0_WIFI 0/10 VDD_IN 8691/9985 VDD_SYS_CPU 3214/2668 VDD_SYS_DDR 1602/2059
[2019-05-23 20:30:58] RAM 4730/7852MB (lfb 238x4MB) CPU [2%@345,1%@345,1%@345,0%@345,0%@345,0%@345] EMC_FREQ 0% GR3D_FREQ 0% PLL@71.5C MCPU@71.5C PMIC@100C Tboard@66C GPU@70.5C BCPU@71.5C thermal@71.7C Tdiode@71C VDD_SYS_GPU 306/2396 VDD_SYS_SOC 1073/1979 VDD_4V0_WIFI 0/10 VDD_IN 3490/9984 VDD_SYS_CPU 690/2668 VDD_SYS_DDR 667/2058
[2019-05-23 20:30:59] RAM 4730/7852MB (lfb 238x4MB) CPU [1%@345,4%@345,1%@345,0%@345,0%@345,0%@345] EMC_FREQ 0% GR3D_FREQ 0% PLL@70.5C MCPU@70.5C PMIC@100C Tboard@66C GPU@69.5C BCPU@70.5C thermal@70.6C Tdiode@70.25C VDD_SYS_GPU 230/2396 VDD_SYS_SOC 843/1979 VDD_4V0_WIFI 0/10 VDD_IN 2609/9984 VDD_SYS_CPU 460/2668 VDD_SYS_DDR 400/2058
[2019-05-23 20:31:00] RAM 4730/7852MB (lfb 238x4MB) CPU [3%@345,6%@345,2%@345,0%@345,1%@345,0%@345] EMC_FREQ 0% GR3D_FREQ 0% PLL@70.5C MCPU@70.5C PMIC@100C Tboard@66C GPU@69C BCPU@70.5C thermal@70.1C Tdiode@69.75C VDD_SYS_GPU 230/2396 VDD_SYS_SOC 767/1979 VDD_4V0_WIFI 0/10 VDD_IN 2493/9983 VDD_SYS_CPU 460/2667 VDD_SYS_DDR 362/2058
[2019-05-23 20:31:01] RAM 4730/7852MB (lfb 238x4MB) CPU [1%@345,0%@345,4%@345,0%@345,0%@345,0%@345] EMC_FREQ 0% GR3D_FREQ 0% PLL@70C MCPU@70C PMIC@100C Tboard@66C GPU@68.5C BCPU@70C thermal@69.4C Tdiode@69.25C VDD_SYS_GPU 230/2396 VDD_SYS_SOC 767/1979 VDD_4V0_WIFI 0/10 VDD_IN 2378/9982 VDD_SYS_CPU 383/2667 VDD_SYS_DDR 324/2058
[2019-05-23 20:31:02] RAM 4730/7852MB (lfb 238x4MB) CPU [2%@345,3%@345,4%@345,0%@345,0%@345,0%@345] EMC_FREQ 0% GR3D_FREQ 0% PLL@69.5C MCPU@69.5C PMIC@100C Tboard@66C GPU@68C BCPU@69.5C thermal@69.1C Tdiode@68.75C VDD_SYS_GPU 230/2396 VDD_SYS_SOC 690/1979 VDD_4V0_WIFI 0/10 VDD_IN 2340/9981 VDD_SYS_CPU 383/2667 VDD_SYS_DDR 305/2058
[2019-05-23 20:31:03] RAM 4730/7852MB (lfb 238x4MB) CPU [4%@345,4%@345,2%@345,0%@345,0%@345,0%@345] EMC_FREQ 0% GR3D_FREQ 0% PLL@69.5C MCPU@69.5C PMIC@100C Tboard@66C GPU@68C BCPU@69.5C thermal@68.9C Tdiode@68.5C VDD_SYS_GPU 230/2395 VDD_SYS_SOC 767/1979 VDD_4V0_WIFI 324/10 VDD_IN 2762/9980 VDD_SYS_CPU 383/2667 VDD_SYS_DDR 343/2057
Chunk of kernel log containing crash
May 23 20:30:56 nvidia-tx2-2 kernel: [10241.557539] tegradc 15210000.nvdisplay: hdmi: tmds rate:148500K prod-setting:prod_c_hdmi_111m_223m
May 23 20:30:56 nvidia-tx2-2 kernel: [10241.558934] tegradc 15210000.nvdisplay: hdmi: get RGB quant from EDID.
May 23 20:30:56 nvidia-tx2-2 kernel: [10241.558964] tegradc 15210000.nvdisplay: hdmi: get YCC quant from EDID.
May 23 20:30:56 nvidia-tx2-2 kernel: [10241.562648] extcon-disp-state external-connection:disp-state: cable 47 state 1
May 23 20:30:56 nvidia-tx2-2 kernel: [10241.562650] Extcon AUX1(HDMI) enable
May 23 20:30:56 nvidia-tx2-2 kernel: [10241.565113] tegradc 15210000.nvdisplay: unblank
May 23 20:30:56 nvidia-tx2-2 kernel: [10241.677729] nvgpu: 17000000.gp10b __nvgpu_timeout_expired_msg_cpu:94 [ERR] Timeout detected @ nvgpu_vm_unmap+0xe0/0x188 [nvgpu] sync-unmap failed on 0x1e27000000
May 23 20:30:56 nvidia-tx2-2 kernel: [10241.693100] nvgpu: 17000000.gp10b __nvgpu_timeout_expired_msg_cpu:94 [ERR] Timeout detected @ nvgpu_vm_unmap+0x118/0x188 [nvgpu]
May 23 20:30:57 nvidia-tx2-2 kernel: [10241.747921] ------------[ cut here ]------------
May 23 20:30:57 nvidia-tx2-2 kernel: [10241.747929] kernel BUG at /dvs/git/dirty/git-master_linux/kernel/nvgpu/drivers/gpu/nvgpu/common/mm/vm.c:774!
May 23 20:30:57 nvidia-tx2-2 kernel: [10241.757757] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
May 23 20:30:57 nvidia-tx2-2 kernel: [10241.763237] Modules linked in: binfmt_misc bnep fuse bcmdhd nvs_bmi160 nvs cfg80211 nvgpu bluedroid_pm ip_tables x_tables
May 23 20:30:57 nvidia-tx2-2 kernel: [10241.774330] CPU: 1 PID: 6684 Comm: Xorg Not tainted 4.9.140-tegra #1
May 23 20:30:57 nvidia-tx2-2 kernel: [10241.780674] Hardware name: quill (DT)
May 23 20:30:57 nvidia-tx2-2 kernel: [10241.784335] task: ffffffc1b98be200 task.stack: ffffffc1bd46c000
May 23 20:30:57 nvidia-tx2-2 kernel: [10241.790466] PC is at nvgpu_vm_get_buffers+0x10c/0x128 [nvgpu]
May 23 20:30:57 nvidia-tx2-2 kernel: [10241.796375] LR is at nvgpu_vm_get_buffers+0xc8/0x128 [nvgpu]
May 23 20:30:57 nvidia-tx2-2 kernel: [10241.802030] pc : [<ffffff8000fab204>] lr : [<ffffff8000fab1c0>] pstate: 20400145
May 23 20:30:57 nvidia-tx2-2 kernel: [10241.809414] sp : ffffffc1bd46fba0
May 23 20:30:57 nvidia-tx2-2 kernel: [10241.812725] x29: ffffffc1bd46fba0 x28: ffffffc1e23c8000
May 23 20:30:57 nvidia-tx2-2 kernel: [10241.818053] x27: ffffffc16a54b680 x26: 0000000000000001
May 23 20:30:57 nvidia-tx2-2 kernel: [10241.823378] x25: 0000000000000000 x24: ffffffc1cf647858
May 23 20:30:57 nvidia-tx2-2 kernel: [10241.828702] x23: ffffffc1bd46fc34 x22: ffffffc1bd46fc38
May 23 20:30:57 nvidia-tx2-2 kernel: [10241.834025] x21: ffffffc083233000 x20: ffffffc1cf647800
May 23 20:30:57 nvidia-tx2-2 kernel: [10241.839351] x19: 0000000000000115 x18: 0000000000002077
May 23 20:30:57 nvidia-tx2-2 kernel: [10241.844675] x17: 000000000000009d x16: 0000000000000000
May 23 20:30:57 nvidia-tx2-2 kernel: [10241.850000] x15: 00002530e0000000 x14: 0018484730000000
May 23 20:30:57 nvidia-tx2-2 kernel: [10241.855324] x13: 000000005ce73b40 x12: 0000000000000017
May 23 20:30:57 nvidia-tx2-2 kernel: [10241.860649] x11: 0000000025d201c0 x10: 0000000000002801
May 23 20:30:57 nvidia-tx2-2 kernel: [10241.865974] x9 : 0000000000000000 x8 : ffffffc083234000
May 23 20:30:57 nvidia-tx2-2 kernel: [10241.871299] x7 : 00000000024000c0 x6 : ffffffc083233000
May 23 20:30:57 nvidia-tx2-2 kernel: [10241.876623] x5 : 00000000000008b0 x4 : ffffff8001060bc8
May 23 20:30:57 nvidia-tx2-2 kernel: [10241.881947] x3 : 0000000000000062 x2 : 0000000000000000
May 23 20:30:57 nvidia-tx2-2 kernel: [10241.887271] x1 : 0000001e2b000000 x0 : 0000000000000116
May 23 20:30:57 nvidia-tx2-2 kernel: [10241.892596]
May 23 20:30:57 nvidia-tx2-2 kernel: [10241.894090] Process Xorg (pid: 6684, stack limit = 0xffffffc1bd46c000)
May 23 20:30:57 nvidia-tx2-2 kernel: [10241.900606] Call trace:
May 23 20:30:57 nvidia-tx2-2 kernel: [10241.903261] [<ffffff8000fab204>] nvgpu_vm_get_buffers+0x10c/0x128 [nvgpu]
May 23 20:30:57 nvidia-tx2-2 kernel: [10241.910215] [<ffffff8000fc5738>] gk20a_channel_add_job+0x50/0x160 [nvgpu]
May 23 20:30:57 nvidia-tx2-2 kernel: [10241.917166] [<ffffff8000fc7124>] gk20a_channel_semaphore_wakeup+0xa2c/0xee8 [nvgpu]
May 23 20:30:57 nvidia-tx2-2 kernel: [10241.924983] [<ffffff8000fc764c>] nvgpu_submit_channel_gpfifo_user+0x6c/0x88 [nvgpu]
May 23 20:30:57 nvidia-tx2-2 kernel: [10241.932794] [<ffffff8000f7ee88>] gk20a_channel_ioctl+0x728/0x1280 [nvgpu]
May 23 20:30:57 nvidia-tx2-2 kernel: [10241.939599] [<ffffff8008271a90>] do_vfs_ioctl+0xb0/0x8d8
May 23 20:30:57 nvidia-tx2-2 kernel: [10241.944907] [<ffffff8008272344>] SyS_ioctl+0x8c/0xa8
May 23 20:30:57 nvidia-tx2-2 kernel: [10241.949888] [<ffffff80080838c0>] el0_svc_naked+0x34/0x38
May 23 20:30:57 nvidia-tx2-2 kernel: [10241.955194] ---[ end trace ce78e6d774e3f220 ]---
May 23 21:08:08 nvidia-tx2-2 kernel: [12473.724186] usb 1-2.2: USB disconnect, device number 4
May 23 21:08:10 nvidia-tx2-2 kernel: [12475.516220] usb 1-2.1: USB disconnect, device number 3
May 23 21:08:10 nvidia-tx2-2 kernel: [12475.785328] usb 1-2: usb_suspend_both: status 0
May 23 21:08:10 nvidia-tx2-2 kernel: [12475.790112] usb usb1: usb_suspend_both: status 0
May 23 21:08:10 nvidia-tx2-2 kernel: [12475.794891] tegra-xusb 3530000.xhci: entering ELPG
May 23 21:08:10 nvidia-tx2-2 kernel: [12475.804572] tegra-xusb 3530000.xhci: entering ELPG done
May 23 21:08:11 nvidia-tx2-2 kernel: [12476.595095] tegra-xusb 3530000.xhci: exiting ELPG
May 23 21:08:11 nvidia-tx2-2 kernel: [12476.603557] tegra-xusb 3530000.xhci: Firmware timestamp: 2018-12-26 10:30:04 UTC, Version: 55.12 release
May 23 21:08:11 nvidia-tx2-2 kernel: [12476.614207] tegra-xusb 3530000.xhci: exiting ELPG done
May 23 21:08:11 nvidia-tx2-2 kernel: [12476.619709] usb usb1: usb_suspend_both: status -16
May 23 21:08:11 nvidia-tx2-2 kernel: [12476.624925] usb usb1: usb_suspend_both: status -16
May 23 21:08:11 nvidia-tx2-2 kernel: [12476.629802] usb usb1: usb_suspend_both: status -16
May 23 21:08:11 nvidia-tx2-2 kernel: [12476.634890] usb usb1: usb_suspend_both: status -16
May 23 21:08:11 nvidia-tx2-2 kernel: [12476.639745] usb usb1: usb_suspend_both: status -16
May 23 21:08:11 nvidia-tx2-2 kernel: [12476.644593] usb usb1: usb_suspend_both: status -16
May 23 21:08:11 nvidia-tx2-2 kernel: [12476.649450] usb usb1: usb_suspend_both: status -16
May 23 21:08:11 nvidia-tx2-2 kernel: [12476.654305] usb usb1: usb_suspend_both: status -16
May 23 21:08:11 nvidia-tx2-2 kernel: [12476.659288] usb usb1: usb_suspend_both: status -16
May 23 21:08:11 nvidia-tx2-2 kernel: [12476.869320] usb 1-2.4: new low-speed USB device number 5 using tegra-xusb
May 23 21:08:12 nvidia-tx2-2 kernel: [12476.913817] usb 1-2.4: New USB device found, idVendor=413c, idProduct=2105
May 23 21:08:12 nvidia-tx2-2 kernel: [12476.920819] usb 1-2.4: New USB device strings: Mfr=1, Product=2, SerialNumber=0
May 23 21:08:12 nvidia-tx2-2 kernel: [12476.928210] usb 1-2.4: Product: Dell USB Keyboard
May 23 21:08:12 nvidia-tx2-2 kernel: [12476.932975] usb 1-2.4: Manufacturer: Dell
May 23 21:08:12 nvidia-tx2-2 kernel: [12476.956545] input: Dell Dell USB Keyboard as /devices/3530000.xhci/usb1/1-2/1-2.4/1-2.4:1.0/0003:413C:2105.0003/input/input5
May 23 21:08:12 nvidia-tx2-2 kernel: [12477.026557] hid-generic 0003:413C:2105.0003: input,hidraw0: USB HID v1.10 Keyboard [Dell Dell USB Keyboard] on usb-3530000.xhci-2.4/input0
May 23 21:08:14 nvidia-tx2-2 kernel: [12479.397318] usb 1-2.1: new low-speed USB device number 6 using tegra-xusb
May 23 21:08:14 nvidia-tx2-2 kernel: [12479.426189] usb 1-2.1: New USB device found, idVendor=04f2, idProduct=0939
May 23 21:08:14 nvidia-tx2-2 kernel: [12479.433256] usb 1-2.1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
May 23 21:08:14 nvidia-tx2-2 kernel: [12479.440675] usb 1-2.1: Product: USB Optical Mouse