why something killed my program?

Autonomous Machines Jetson & Embedded Systems Jetson AGX Xavier
#1

my program was killed after a few moment without any operation.is there something kill my programe because of xavier’s power consumption?
there are xavier’s statistic:
GPU highest consumption:15000/6000
GR3D_FREQ highest to 99%

CPU [9%@2265,11%@2265,6%@2265,3%@2265,12%@2265,29%@2265,17%@2265,17%@2265] EMC_FREQ 0% GR3D_FREQ 64% AO@35.5C GPU@42.5C Tboard@35C Tdiode@40.25C AUX@35C CPU@39.5C thermal@36.5C PMIC@100C GPU 14824/4893 CPU 1833/1788 SOC 6880/4588 CV 0/0 VDDRQ 2292/1012 SYS5V 3400/2845

and what GPU 14824/4893 means? (Current GPU power consumption/Average GPU power consumption) Is the unit of power consumption watt？

#2 
RAM 12400/15819MB (lfb 304x4MB) CPU [8%@2265,9%@2265,30%@2265,6%@2265,7%@2265,26%@2265,3%@2265,4%@2265] EMC_FREQ 0% GR3D_FREQ 1% AO@35.5C GPU@38.5C Tboard@35C Tdiode@40.25C AUX@35.5C CPU@38.5C thermal@38.15C PMIC@100C GPU 11792/4962 CPU 1991/1803 SOC 6131/4612 CV 0/0 VDDRQ 1838/1021 SYS5V 3200/2849
RAM 12401/15819MB (lfb 304x4MB) CPU [9%@1190,19%@1190,9%@1190,2%@1190,5%@1190,5%@1190,41%@1190,25%@1190] EMC_FREQ 0% GR3D_FREQ 99% AO@35C GPU@39C Tboard@35C Tdiode@39.5C AUX@35C CPU@38.5C thermal@37.1C PMIC@100C GPU 5839/4972 CPU 2152/1807 SOC 5690/4625 CV 0/0 VDDRQ 1383/1025 SYS5V 3044/2851
RAM 12682/15819MB (lfb 292x4MB) CPU [10%@2265,21%@2265,6%@2265,6%@2265,11%@2265,31%@2265,8%@2265,18%@2265] EMC_FREQ 0% GR3D_FREQ 1% AO@35C GPU@37.5C Tboard@35C Tdiode@39.75C AUX@35C CPU@39.5C thermal@38.5C PMIC@100C GPU 5687/4981 CPU 2459/1815 SOC 5228/4632 CV 0/0 VDDRQ 1076/1026 SYS5V 2924/2852
RAM 12682/15819MB (lfb 292x4MB) CPU [9%@1190,20%@1190,14%@1190,5%@1190,4%@1190,17%@1190,12%@1190,22%@2265] EMC_FREQ 0% GR3D_FREQ 2% AO@36C GPU@40C Tboard@35C Tdiode@40.5C AUX@35.5C CPU@39.5C thermal@38.6C PMIC@100C GPU 15130/5102 CPU 1834/1815 SOC 6730/4657 CV 0/0 VDDRQ 2140/1039 SYS5V 3320/2858
RAM 12861/15819MB (lfb 292x4MB) CPU [11%@1190,10%@1190,4%@1190,5%@1190,19%@1190,28%@1190,18%@1190,12%@1190] EMC_FREQ 0% GR3D_FREQ 99% AO@36.5C GPU@44.5C Tboard@35C Tdiode@40.5C AUX@35.5C CPU@39.5C thermal@39.25C PMIC@100C GPU 19208/5268 CPU 1525/1812 SOC 7018/4685 CV 0/0 VDDRQ 2592/1058 SYS5V 3560/2866
RAM 12862/15819MB (lfb 292x4MB) CPU [8%@2265,30%@2265,7%@2265,22%@2265,14%@2265,12%@2265,4%@2265,33%@2265] EMC_FREQ 0% GR3D_FREQ 84% AO@35.5C GPU@37C Tboard@35C Tdiode@39.75C AUX@35C CPU@40C thermal@37.3C PMIC@100C GPU 2771/5239 CPU 2925/1825 SOC 4775/4686 CV 0/0 VDDRQ 923/1056 SYS5V 2804/2865
RAM 12947/15819MB (lfb 292x4MB) CPU [8%@2265,6%@2265,27%@2265,9%@2265,6%@2265,9%@2265,4%@2265,20%@2265] EMC_FREQ 0% GR3D_FREQ 2% AO@36C GPU@40.5C Tboard@35C Tdiode@40.5C AUX@35C CPU@39C thermal@37.3C PMIC@100C GPU 15289/5354 CPU 1681/1823 SOC 6271/4704 CV 0/0 VDDRQ 2140/1068 SYS5V 3360/2871
RAM 12948/15819MB (lfb 292x4MB) CPU [14%@2265,6%@2265,3%@2265,13%@2265,6%@2265,17%@2265,26%@2265,39%@2265] EMC_FREQ 0% GR3D_FREQ 6% AO@35C GPU@37.5C Tboard@35C Tdiode@39.75C AUX@35.5C CPU@39.5C thermal@37.9C PMIC@100C GPU 3386/5332 CPU 2462/1830 SOC 5081/4708 CV 0/0 VDDRQ 1077/1069 SYS5V 2884/2871
RAM 13118/15819MB (lfb 276x4MB) CPU [7%@1190,6%@1190,11%@1190,20%@1190,6%@1190,19%@1190,15%@1190,13%@1190] EMC_FREQ 0% GR3D_FREQ 99% AO@36C GPU@43.5C Tboard@35C Tdiode@40.5C AUX@35.5C CPU@40C thermal@37.15C PMIC@100C GPU 14977/5440 CPU 1528/1827 SOC 7033/4734 CV 0/0 VDDRQ 2292/1082 SYS5V 3400/2877
RAM 13119/15819MB (lfb 276x4MB) CPU [10%@2265,31%@2265,9%@2265,27%@2265,1%@2265,33%@2265,4%@2265,7%@2265] EMC_FREQ 0% GR3D_FREQ 99% AO@35C GPU@38C Tboard@35C Tdiode@39.75C AUX@35.5C CPU@39.5C thermal@37.6C PMIC@100C GPU 3846/5422 CPU 2615/1836 SOC 5081/4738 CV 0/0 VDDRQ 1076/1082 SYS5V 2884/2877
RAM 13397/15819MB (lfb 207x4MB) CPU [23%@2265,7%@2265,4%@2265,10%@2265,9%@2265,19%@2265,13%@2265,10%@2265] EMC_FREQ 0% GR3D_FREQ 1% AO@35.5C GPU@38.5C Tboard@35C Tdiode@40.25C AUX@35.5C CPU@39.5C thermal@38.2C PMIC@100C GPU 10269/5476 CPU 2300/1841 SOC 5676/4748 CV 0/0 VDDRQ 1686/1089 SYS5V 3124/2880
#3

hello quanminzhu,

your comment #2 shows the system memory usage is continuously increasing.
one condition might be watch dog killer kick-in to force stop processes when there’s not enough system memory.
there will kernel message pop-up if your process shutdown by watch dog killer,
could you please connect serial port and check kernel messages for details.
thanks

#4

thanks for your reply.could
how to connect serial port and check kernel messages for details?
and is there any way to konw gpu memory?

#5

For serial console, if you connect the micro-B USB cable to your host, and monitor “dmesg --follow” during insert, then you should see a serial FTDI UART mentioned. Most likely it will appear as “/dev/ttyUSB0”, but adjust for the tty name.

Then install something like gtkterm and it becomes a console if started with the correct arguments. Assuming “ttyUSB0”:

gtkterm -b 8 -t 1 -s 115200 -p /dev/ttyUSB0

(8 bits, 1 stop bit, no parity, speed 115200)

#6