Jetson nano random shutdown, not a power issue

Hi,
Jetson Development Team.
Thanks for your great support.

I came across a new problem which I tried a lot but was not able to fix.
In my project there are 3 Jetson nano camera systems. Each one has a dedicated power management board (dc-dc converter, voltage, current monitoring etc).

Out of 3 systems, one is facing an unexpected shutdown. Sometimes it used to run for 3-4 hours and sometimes for 18-19 hours before getting shut down.

It’s not a power issue. I saw my voltage current monitor log. The last log timestamp is 19:43 (17/06/2022-19:43:05,T=29.90,H=64,I=0.562,V=11.75,P=6.60). I didn’t found any error/warning at 19:43 or near around. I am attaching here all the logs for your reference.

Please look into the issue and help me.

Thanks

Suman

kern.log (3.3 MB)
syslog (819.4 KB)
VoltageCurrentTemperature_log.txt (585.2 KB)

Sorry if this is totally misguided.

My experience with standard CSI cameras is that
camera sudden shutdown ghost
The first thing that comes to my mind from these keywords is poor contact between the cable and the connector.
Can this sudden shutdown be reproduced by intentionally pulling out the camera cable?

I have experienced shutdowns by shaking the cable, touching it, and sometimes just getting close to it.
The ghost that thrilled me was exorcised with sellotape.

Now it is fixed with caulking.

Thanks Naisy for the response. My case is completely different. All my power connectors are industrial grade, and my 3 camera systems are identical, but one shutdown occurs randomly. There is no pattern.

Hi
Jetson Development Team,
Please anyone look into the issue and help me.

I am waiting for the response
Thanks

Have you tried to confirm that’s camera issue, Nano module or carrier board problem by swaping in different combination?

Thanks Kayccc for the response, I think there is no issue with the camera. Camera is working perfectly. I am performing object detection inference and it is working sometime for 18-19 hours also and some time within 2-3 hours it used to shutdown. I checked for power disturbance, but all okay. You can see log.

In all my system I am using SSD. I am running jetpack 4.6.

Jun 17 19:26:54 visnode_TX_Top kernel: [   33.906126] vdd-fan: disabling
Jun 17 19:26:54 visnode_TX_Top kernel: [   33.906130] vdd-usb-vbus: disabling
Jun 17 19:26:54 visnode_TX_Top kernel: [   33.906133] vdd-usb-vbus2: disabling
Jun 17 19:26:54 visnode_TX_Top kernel: [   33.906144] vddio-sdmmc3-ap: disabling
Jun 17 19:26:54 visnode_TX_Top kernel: [   33.906217] vdd-3v3-sd: disabling
Jun 17 19:26:54 visnode_TX_Top kernel: [   33.906219] avdd-io-edp-1v05: disabling
Jun 17 19:26:54 visnode_TX_Top kernel: [   33.906222] vdd-usb-hub-en: disabling

If you see syslog, there is a message show above, Is this message are normal?

Please help me, my project deadline is coming very close

Thanks
Suman

Those logs are normal. That is to save power if some regulators are not in use.

I would suggest you give out log so that we can see the issue more clearly.

Is your issue “random shutdown” or “random reboot”?

A random shutdown issue is more related to hardware while a random reboot may be more closed to software.

Hi Wayne,
My issue is “random shutdown”, I understand it can be hardware issue. But is there a way to detect from log. I gues jetson nano dev kit has its own power monitoring on the hardware level. (under/over voltage /or high current monitoring)
Do jetson nano save any power warning message in any log? If so please let me know, I am not clear what is the root cause.

Thanks
Suman

You can run tegrastats application in the uart serial console check if the last power consumption before shutdown is abnormal.

This is custom carrier board, right?

Okay thanks I will give a try to tegrastats app and I am using official dev kit carrier board B01. It is running 24/7 and kept inside industrial grade enclosure with proper forced air circulation.

If this is devkit, have you checked whether the power adapter spec is matching what we suggested?

Yes Wayne, I am using it as per suggestion. I have on board 5V 5A DC-DC converter which takes 12V input and convert it into 5V.

Hi Wayne,
I am attaching tegrastats output, please have a look.

suman@visnode_TX_Top:~$ tegrastats
RAM 1022/3964MB (lfb 477x4MB) SWAP 0/6000MB (cached 0MB) CPU [6%@1479,8%@1479,13%@1479,11%@1479] EMC_FREQ 0% GR3D_FREQ 0% PLL@31C CPU@32.5C PMIC@50C GPU@31.5C AO@39.5C thermal@31.75C
RAM 1022/3964MB (lfb 477x4MB) SWAP 0/6000MB (cached 0MB) CPU [10%@1479,5%@1479,10%@1479,9%@1479] EMC_FREQ 0% GR3D_FREQ 0% PLL@30.5C CPU@32C PMIC@50C GPU@31.5C AO@39C thermal@31.5C
RAM 1022/3964MB (lfb 477x4MB) SWAP 0/6000MB (cached 0MB) CPU [19%@1479,19%@1479,20%@1479,17%@1479] EMC_FREQ 0% GR3D_FREQ 0% PLL@31C CPU@32C PMIC@50C GPU@31.5C AO@40C thermal@31.5C
RAM 1031/3964MB (lfb 477x4MB) SWAP 0/6000MB (cached 0MB) CPU [12%@1479,24%@1479,15%@1479,14%@1479] EMC_FREQ 0% GR3D_FREQ 0% PLL@31C CPU@32C PMIC@50C GPU@31.5C AO@40C thermal@32.25C
RAM 1031/3964MB (lfb 477x4MB) SWAP 0/6000MB (cached 0MB) CPU [8%@1479,5%@1479,8%@1479,11%@1479] EMC_FREQ 0% GR3D_FREQ 0% PLL@31C CPU@31.5C PMIC@50C GPU@31C AO@40C thermal@32C
RAM 1031/3964MB (lfb 477x4MB) SWAP 0/6000MB (cached 0MB) CPU [10%@1479,5%@1479,2%@1479,6%@1479] EMC_FREQ 0% GR3D_FREQ 0% PLL@30.5C CPU@32C PMIC@50C GPU@31.5C AO@39.5C thermal@31.75C
RAM 1031/3964MB (lfb 477x4MB) SWAP 0/6000MB (cached 0MB) CPU [8%@1479,7%@1479,6%@1479,9%@1479] EMC_FREQ 0% GR3D_FREQ 0% PLL@30.5C CPU@32C PMIC@50C GPU@31.5C AO@40C thermal@31.75C
RAM 1031/3964MB (lfb 477x4MB) SWAP 0/6000MB (cached 0MB) CPU [9%@1479,8%@1479,6%@1479,5%@1479] EMC_FREQ 0% GR3D_FREQ 0% PLL@30.5C CPU@31.5C PMIC@50C GPU@32C AO@39.5C thermal@31.75C
RAM 1031/3964MB (lfb 477x4MB) SWAP 0/6000MB (cached 0MB) CPU [6%@1479,5%@1479,6%@1479,5%@1479] EMC_FREQ 0% GR3D_FREQ 0% PLL@30.5C CPU@31.5C PMIC@50C GPU@31.5C AO@40C thermal@31.5C
RAM 1031/3964MB (lfb 477x4MB) SWAP 0/6000MB (cached 0MB) CPU [4%@1479,5%@1479,5%@1479,7%@1479] EMC_FREQ 0% GR3D_FREQ 0% PLL@31C CPU@31.5C PMIC@50C GPU@32C AO@40C thermal@31.75C
RAM 1031/3964MB (lfb 477x4MB) SWAP 0/6000MB (cached 0MB) CPU [7%@1479,5%@1479,6%@1479,5%@1479] EMC_FREQ 0% GR3D_FREQ 0% PLL@30.5C CPU@31.5C PMIC@50C GPU@32C AO@39.5C thermal@31.75C
RAM 1031/3964MB (lfb 477x4MB) SWAP 0/6000MB (cached 0MB) CPU [6%@1479,4%@1479,5%@1479,5%@1479] EMC_FREQ 0% GR3D_FREQ 0% PLL@31C CPU@32C PMIC@50C GPU@31.5C AO@40C thermal@31.75C
RAM 1031/3964MB (lfb 477x4MB) SWAP 0/6000MB (cached 0MB) CPU [7%@1479,5%@1479,6%@1479,6%@1479] EMC_FREQ 0% GR3D_FREQ 0% PLL@30.5C CPU@31.5C PMIC@50C GPU@31.5C AO@39.5C thermal@31.75C
RAM 1031/3964MB (lfb 477x4MB) SWAP 0/6000MB (cached 0MB) CPU [6%@1479,5%@1479,2%@1479,5%@1479] EMC_FREQ 0% GR3D_FREQ 0% PLL@30.5C CPU@31.5C PMIC@50C GPU@31.5C AO@39.5C thermal@31.75C
RAM 1031/3964MB (lfb 477x4MB) SWAP 0/6000MB (cached 0MB) CPU [5%@1479,5%@1479,5%@1479,4%@1479] EMC_FREQ 0% GR3D_FREQ 0% PLL@30.5C CPU@31.5C PMIC@50C GPU@31.5C AO@40C thermal@31.5C
RAM 1032/3964MB (lfb 477x4MB) SWAP 0/6000MB (cached 0MB) CPU [10%@1479,7%@1479,6%@1479,9%@1479] EMC_FREQ 0% GR3D_FREQ 0% PLL@31C CPU@31.5C PMIC@50C GPU@31.5C AO@40C thermal@31.75C
RAM 1032/3964MB (lfb 477x4MB) SWAP 0/6000MB (cached 0MB) CPU [5%@1479,5%@1479,5%@1479,6%@1479] EMC_FREQ 0% GR3D_FREQ 0% PLL@30.5C CPU@32C PMIC@50C GPU@31.5C AO@40C thermal@32C
RAM 1032/3964MB (lfb 477x4MB) SWAP 0/6000MB (cached 0MB) CPU [8%@1479,7%@1479,3%@1479,5%@1479] EMC_FREQ 0% GR3D_FREQ 0% PLL@31C CPU@32.5C PMIC@50C GPU@32C AO@40C thermal@31.75C
RAM 1032/3964MB (lfb 477x4MB) SWAP 0/6000MB (cached 0MB) CPU [13%@1479,10%@1479,4%@1479,4%@1479] EMC_FREQ 0% GR3D_FREQ 0% PLL@30.5C CPU@32C PMIC@50C GPU@31.5C AO@40C thermal@31.75C

Will it be possible to disable power save mode, so that it will not disable (vdd-usb-vbus: disabling).
As jetson nano is running on USB SSD. If usb power is getting disabled that might bring unexpected shutdown.

Please give response