I can run multiple deepstreem_test_3.py (up to 7 pipeline with each feeding 4 video files) without crash the system (or causing it self reboot) if I disconnect ethernet network on AGX Xavier (by clicking top right corner network icon and select disconnect right below wired connection 1) .
However, if I reconnect the network and run the above same multiple deepstream_test_3.py then the system crash (self reboot).
Steps to duplicate the crash (self reboot):
- turn on AGX and make sure network is connected
- set MAXN mode, set fan at 255
- run 5 to 7 copies of deepstream_test_3.py feeding 4 videos each (the more copy to run the easier to duplicate the problem)
- go to PC (running ubuntu 18.04) and “ssh agx.local” to connect to the AGX and then run tegrastats in background to log the status every second, then use “tail tegralog” to view the log frequently
- around the 5th or 6th copy of deepstream_test_3.py running, the system crash (then self reboot)
Background: it has been a long way to lead to this path. Initially I suspect my power supply voltage swing, so I add a 600W line conditioner to eliminate the power issue. Then I suspect it is thermal issue, but check the tegrastats, the GPU temperature never exceed 47C, of course other CPU, thermal temperature are lower than 47C. Eventually thanks to linuxdev pointed out in one of my self-rebooting logs actually the network causing the self reboot! And this lead to this post of showing how to duplicate the issue. Attached please find the serial console log 7_run4_network_on_crash.log (233.4 KB) and tegrastats log 7_run4_network_on_crash_tegrastats.log (80.6 KB)
when the system crash. Be aware that the network error may not always show up in the console log. But so far whenever the network is on, the system is not stable. I have been changing two different routers, the result is the same: Network on, system very easy crash when running multiple pipelines. Network off, the system is very solid so far.
Question: my product need to turn on ethernet to transmit the result in real time, now whenever the network is on, the system is not stable (kept self rebooting), how can we overcome this issue? Plus WiFi is not a solution for our product. Please help. Thanks a lot in advance.