When I was running the program, I found that orin did not respond. After trying to restart it, I could not find out why it did not respond at that time. How do I debug it?
Which JetPack SW?
AGX Orin devkit or with custom carrier board?
Any log while issue happening?
The platform is the jetson agx orin developer kit，There is no log print when the error occurs, there is no response when pressing any key, it is dead and must be restarted to respond.
(base) xxx@xxx-desktop:~$ cat /etc/nv_tegra_release
# R35 (release), REVISION: 3.1, GCID: 32827747, BOARD: t186ref, EABI: aarch64, DATE: Sun Mar 19 15:19:21 UTC 2023
We don’t know what kind of program you are running so it’s hard to help.
Update to 35.4.1 and see if it gets mitigated.
Here is my program startup script.As you can see in the script, I will end the previous running program before each start, and some are unable to end the program after kill -2 or the end is too slow I use kill -9 to end again.
sleep 1
echo " " | sudo -S ps -aux |grep gst-launch-1.0 |grep -v grep |awk '{print "echo \" \" | sudo -S kill -2 "$2}' | sh
echo " " | sudo -S ./clock.sh --max
sleep 2
echo " " | sudo -S ps -aux |grep gst-launch-1.0 |grep -v grep |awk '{print "echo \" \" | sudo -S kill -9 "$2}' | sh
TIMESTAMPS=`date +%Y%m%d%H%M%S`
mkdir /media/Data/$TIMESTAMPS
chown xxx:xxx /media/Data/$TIMESTAMPS
FILE_HOME_DIR=/media/Data/$TIMESTAMPS
GST_DEBUG_DUMP_DOT_DIR=/media/Data/log gst-launch-1.0 nvcompositor name=comp sink_0::xpos=0 sink_0::ypos=0 sink_0::width=960 sink_0::height=540 sink_1::xpos=960 sink_1::ypos=0 sink_1::width=960 sink_1::height=540 sink_2::xpos=0 sink_2::ypos=540 sink_2::width=960 sink_2::height=540 sink_3::xpos=960 sink_3::ypos=540 sink_3::width=960 sink_3::height=540 ! nvvidconv interpolation-method=4 ! 'video/x-raw(memory:NVMM),width=1920,height=1080,format=NV12,framerate=10/1' ! nvv4l2h264enc iframeinterval=10 insert-sps-pps=true bitrate=$bitrate_preview ! h264parse ! flvmux ! rtmpsink blocksize=24834048 location='rtmp://127.0.0.1/live/mux4' nvarguscamerasrc sensor-id=0 wbmode=0 ee-mode=0 saturation=0 blocksize=99336192 ! 'video/x-raw(memory:NVMM), width=(int)4128, height=(int)3008,format=(string)NV12, framerate=(fraction)10/1' ! tee name=cam0 ! queue ! nvvidconv ! "video/x-raw" ! multifilesink max-file-size=1862553600 next-file=4 blocksize=4096 location=$FILE_HOME_DIR/cam0-%04d.nv12 -e cam0. ! queue ! comp.sink_0 cam0. ! queue ! nvvidconv ! 'video/x-raw(memory:NVMM),width=3840,height=2160,format=NV12,framerate=10/1' ! nvv4l2h265enc bitrate=$shutter_record capture-io-mode=2 maxperf-enable=true ! h265parse ! mp4mux ! filesink location=$FILE_HOME_DIR/cam-0.mp4 nvarguscamerasrc sensor-id=1 wbmode=0 ee-mode=0 saturation=0 blocksize=99336192 ! 'video/x-raw(memory:NVMM), width=(int)4128, height=(int)3008,format=(string)NV12, framerate=(fraction)10/1' ! tee name=cam1 ! queue ! comp.sink_1 nvarguscamerasrc sensor-id=2 wbmode=0 ee-mode=0 saturation=0 blocksize=99336192 ! 'video/x-raw(memory:NVMM), width=(int)4128, height=(int)3008,format=(string)NV12, framerate=(fraction)10/1' ! tee name=cam2 ! queue ! comp.sink_2 nvarguscamerasrc sensor-id=3 wbmode=0 ee-mode=0 saturation=0 blocksize=99336192 ! 'video/x-raw(memory:NVMM), width=(int)4128, height=(int)3008,format=(string)NV12, framerate=(fraction)10/1' ! tee name=cam3 ! queue ! nvvidconv ! "video/x-raw" ! multifilesink max-file-size=1862553600 next-file=4 blocksize=4096 location=$FILE_HOME_DIR/cam3-%04d.nv12 -e cam3. ! queue ! comp.sink_3 cam3. ! queue ! nvvidconv ! 'video/x-raw(memory:NVMM),width=3840,height=2160,format=NV12,framerate=10/1' ! nvv4l2h265enc bitrate=$shutter_record capture-io-mode=2 maxperf-enable=true ! h265parse ! mp4mux ! filesink location=$FILE_HOME_DIR/cam-3.mp4 > /media/Data/$TIMESTAMPS/frame_time.txt &
So you got nothing from serial console log when it freezed?
Did syslog show anything useful?
How to connect the serial port output log, I connected the monitor, mouse and keyboard.
log.zip (1.4 MB)
This is the log I found in /var/log/syslog and /var/log/kern.log after I restarted
Dec 1 10:27:50 xxx-desktop kernel: [ 6678.064918] cpufreq: cpu8,cur:2381000,set:2201600,set ndiv:172
Dec 1 10:27:57 xxx-desktop kernel: [ 6685.058397] cpufreq: cpu0,cur:2331000,set:2201600,set ndiv:172
Dec 1 10:27:59 xxx-desktop kernel: [ 6687.060785] cpufreq: cpu0,cur:2385000,set:2201600,set ndiv:172
Dec 1 10:28:00 xxx-desktop kernel: [ 6688.061651] cpufreq: cpu0,cur:2086000,set:2201600,set ndiv:172
Dec 1 10:28:02 xxx-desktop kernel: [ 6690.065740] cpufreq: cpu4,cur:2351000,set:2201600,set ndiv:172
Dec 1 10:28:02 xxx-desktop kernel: [ 6690.069285] cpufreq: cpu4,cur:2064000,set:2201600,set ndiv:172
Dec 1 10:28:02 xxx-desktop kernel: [ 6690.072949] cpufreq: cpu8,cur:2349000,set:2201600,set ndiv:172
Dec 1 10:28:04 xxx-desktop kernel: [ 6692.081104] cpufreq: cpu8,cur:1988000,set:2201600,set ndiv:172
Dec 1 10:28:05 xxx-desktop kernel: [ 6693.071224] cpufreq: cpu8,cur:2387000,set:2201600,set ndiv:172
Dec 1 10:28:14 xxx-desktop kernel: [ 6702.076000] cpufreq: cpu4,cur:2052000,set:2201600,set ndiv:172
Dec 1 10:28:15 xxx-desktop kernel: [ 6703.080274] cpufreq: cpu8,cur:2069000,set:2201600,set ndiv:172
Dec 1 10:28:16 xxx-desktop kernel: [ 6704.079566] cpufreq: cpu8,cur:2405000,set:2201600,set ndiv:172
Dec 1 10:28:23 xxx-desktop kernel: [ 6711.082088] cpufreq: cpu4,cur:2040000,set:2201600,set ndiv:172
Dec 1 10:28:23 xxx-desktop kernel: [ 6711.083265] cpufreq: cpu8,cur:2357000,set:2201600,set ndiv:172
Dec 1 10:28:24 xxx-desktop kernel: [ 6712.083350] cpufreq: cpu4,cur:2404000,set:2201600,set ndiv:172
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@Mar 28 01:54:10 xxx-desktop systemd[1]: Mounted Huge Pages File System.
Mar 28 01:54:10 xxx-desktop kernel: [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd421]
Mar 28 01:54:10 xxx-desktop kernel: [ 0.000000] Linux version 5.10.104-tegra (root@ubuntu) (aarch64-buildroot-linux-gnu-gcc.br_real (Buildroot 2020.08) 9.3.0, GNU ld (GNU Binutils) 2.33.1) #10 SMP PREEMPT Mon Jul 10 09:13:14 CST 2023
Mar 28 01:54:10 xxx-desktop kernel: [ 0.000000] OF: fdt: memory scan node memory@80000000, reg size 16,
Mar 28 01:54:10 xxx-desktop kernel: [ 0.000000] OF: fdt: - 80000000 , c0000000
Mar 28 01:54:10 xxx-desktop kernel: [ 0.000000] Machine model: Jetson AGX Orin
Mar 28 01:54:10 xxx-desktop kernel: [ 0.000000] efi: EFI v2.70 by EDK II
See if you can capture something indicating the error from serial cosole:
I set up minicom according to the connection you gave. I used my PC Ubuntu to connect, which is the one in the screenshot below, and directly entered orin, not port printing. I restarted orin without any debug log output.
The connection has to be set up before the device turns on, or you won’t see any log.
To be more detailed, if the orin is not powered on before I connect to the PC, there is no response, wait until the orin is powered on and establish a connection to directly enter the orin.
Hi,
我其實看不太懂你在說什麼，我的意思是你要在開機前就先把UART連線開好才看得到開機階段的log
但其實這也不是重點，重點是你能不能透過serial console在機器crash的時候抓到什麼log
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.