Deepstream pose Estimation Output log "Killed"

virajhapaliya · July 13, 2021, 9:42am

• Hardware Platform (Jetson / GPU) : Jetson Nano
• DeepStream Version : 5.1
• JetPack Version (valid for Jetson only) : 4.5.1
• TensorRT Version : 7.1.3.0

Hello,

Currently, I am working on GitHub - NVIDIA-AI-IOT/deepstream_pose_estimation: This is a sample DeepStream application to demonstrate a human pose estimation pipeline.. I have followed all the steps provided in the repo. I have generated the resnet18_baseline_att_224x224_A_epoch_249.onnx from resnet18_baseline_att_224x224_A_epoch_249.pth

After that from $DEEPSTREAM_DIR/sources/apps/sample_apps/deepstream_pose_estimation I ran
sudo ./deepstream-pose-estimation-app file:///home/amnt_ssip/Downloads/out.h264 out/

After that on the output terminal message was killed. (no other error logs). So I have tried the pose_estimation.onnx model available in the deepstream_pose_estimation folder. And the same message on Terminal.

I have also tried with export GST_DEBUG=6 before running the command to print the all logs on the terminal but it’s not working.

Thank you
Viraj

AastaLLL · July 14, 2021, 2:35am

Hi,

Usually ‘killed’ is caused by the out-of-memory.
Could you monitor the device status at another console simultaneously?

$ sudo tegrastats

Thanks.

virajhapaliya · July 14, 2021, 4:56am

I have check with this command.

Here Are the logs.

RAM 1317/3964MB (lfb 135x4MB) SWAP 831/1982MB (cached 41MB) IRAM 0/252kB(lfb 252kB) CPU [21%@1224,37%@1224,7%@1224,9%@1224] EMC_FREQ 4%@1600 GR3D_FREQ 0%@230 VIC_FREQ 0%@192 APE 25 PLL@26C CPU@30.5C PMIC@100C GPU@26.5C AO@38C thermal@28.5C POM_5V_IN 2628/2927 POM_5V_GPU 78/68 POM_5V_CPU 470/909
RAM 1438/3964MB (lfb 135x4MB) SWAP 831/1982MB (cached 41MB) IRAM 0/252kB(lfb 252kB) CPU [7%@1479,29%@1479,29%@1479,28%@1479] EMC_FREQ 5%@1600 GR3D_FREQ 2%@230 VIC_FREQ 0%@192 APE 25 PLL@27C CPU@30.5C PMIC@100C GPU@26.5C AO@37.5C thermal@28.5C POM_5V_IN 3089/2930 POM_5V_GPU 78/68 POM_5V_CPU 1055/912
RAM 1498/3964MB (lfb 135x4MB) SWAP 831/1982MB (cached 41MB) IRAM 0/252kB(lfb 252kB) CPU [11%@1479,11%@1479,35%@1479,5%@1479] EMC_FREQ 4%@1600 GR3D_FREQ 0%@230 VIC_FREQ 0%@192 APE 25 PLL@26.5C CPU@31.5C PMIC@100C GPU@26.5C AO@37.5C thermal@28.5C POM_5V_IN 2976/2931 POM_5V_GPU 78/68 POM_5V_CPU 1016/913
RAM 1861/3964MB (lfb 135x4MB) SWAP 831/1982MB (cached 41MB) IRAM 0/252kB(lfb 252kB) CPU [15%@1479,14%@1479,51%@1479,7%@1479] EMC_FREQ 5%@1600 GR3D_FREQ 0%@230 VIC_FREQ 0%@192 APE 25 PLL@27C CPU@31.5C PMIC@100C GPU@27C AO@38C thermal@28.5C POM_5V_IN 3167/2935 POM_5V_GPU 78/68 POM_5V_CPU 1094/917
RAM 3515/3964MB (lfb 76x4MB) SWAP 831/1982MB (cached 39MB) IRAM 0/252kB(lfb 252kB) CPU [7%@1479,5%@1479,100%@1479,72%@1479] EMC_FREQ 7%@1600 GR3D_FREQ 0%@76 VIC_FREQ 0%@192 APE 25 PLL@28C CPU@32C PMIC@100C GPU@30C AO@38C thermal@28.75C POM_5V_IN 3891/2952 POM_5V_GPU 0/67 POM_5V_CPU 1942/935
RAM 3562/3964MB (lfb 76x4MB) SWAP 909/1982MB (cached 17MB) IRAM 0/252kB(lfb 252kB) CPU [13%@1479,10%@1479,100%@1479,100%@1479] EMC_FREQ 7%@1600 GR3D_FREQ 0%@76 VIC_FREQ 0%@192 APE 25 PLL@28C CPU@31.5C PMIC@100C GPU@30C AO@37.5C thermal@31C POM_5V_IN 4336/2976 POM_5V_GPU 0/66 POM_5V_CPU 2129/955
RAM 3631/3964MB (lfb 59x4MB) SWAP 1030/1982MB (cached 17MB) IRAM 0/252kB(lfb 252kB) CPU [10%@1479,12%@1479,100%@1479,100%@1479] EMC_FREQ 6%@1600 GR3D_FREQ 0%@76 VIC_FREQ 0%@192 APE 25 PLL@28C CPU@32.5C PMIC@100C GPU@30C AO@38.5C thermal@31C POM_5V_IN 4266/2998 POM_5V_GPU 0/65 POM_5V_CPU 2090/974
RAM 3763/3964MB (lfb 23x4MB) SWAP 1251/1982MB (cached 17MB) IRAM 0/252kB(lfb 252kB) CPU [67%@1479,70%@1479,86%@1479,89%@1479] EMC_FREQ 6%@1600 GR3D_FREQ 0%@76 VIC_FREQ 0%@192 APE 25 PLL@28.5C CPU@32.5C PMIC@100C GPU@27.5C AO@38.5C thermal@29.75C POM_5V_IN 4970/3031 POM_5V_GPU 115/66 POM_5V_CPU 2692/1003
RAM 3792/3964MB (lfb 18x4MB) SWAP 1306/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [11%@1479,61%@1479,100%@1479,25%@1479] EMC_FREQ 5%@1600 GR3D_FREQ 0%@76 VIC_FREQ 0%@192 APE 25 PLL@27.5C CPU@32C PMIC@100C GPU@29.5C AO@38.5C thermal@30.75C POM_5V_IN 3468/3038 POM_5V_GPU 0/65 POM_5V_CPU 1403/1010
RAM 3852/3964MB (lfb 8x4MB) SWAP 1464/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [13%@1479,65%@1479,100%@1479,26%@1479] EMC_FREQ 4%@1600 GR3D_FREQ 0%@76 VIC_FREQ 0%@192 APE 25 PLL@28C CPU@32C PMIC@100C GPU@30C AO@39C thermal@31C POM_5V_IN 4149/3056 POM_5V_GPU 0/64 POM_5V_CPU 2013/1026
RAM 3867/3964MB (lfb 4x4MB) SWAP 1501/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [100%@1479,9%@1479,100%@1479,4%@1479] EMC_FREQ 4%@1600 GR3D_FREQ 0%@76 VIC_FREQ 0%@192 APE 25 PLL@28.5C CPU@32C PMIC@100C GPU@30.5C AO@39C thermal@30.75C POM_5V_IN 4079/3072 POM_5V_GPU 0/62 POM_5V_CPU 2016/1041
RAM 3883/3964MB (lfb 14x2MB) SWAP 1542/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [100%@1479,51%@1479,100%@1479,32%@1479] EMC_FREQ 3%@1600 GR3D_FREQ 0%@76 VIC_FREQ 0%@192 APE 25 PLL@28C CPU@32.5C PMIC@100C GPU@27.5C AO@39C thermal@30C POM_5V_IN 4669/3097 POM_5V_GPU 38/62 POM_5V_CPU 2508/1064
RAM 3888/3964MB (lfb 14x2MB) SWAP 1543/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [100%@1479,98%@1479,100%@1479,95%@1479] EMC_FREQ 3%@1600 GR3D_FREQ 0%@76 VIC_FREQ 0%@192 APE 25 PLL@29C CPU@32.5C PMIC@100C GPU@31C AO@39C thermal@32C POM_5V_IN 5085/3127 POM_5V_GPU 0/61 POM_5V_CPU 3110/1096
RAM 857/3964MB (lfb 89x4MB) SWAP 872/1982MB (cached 3MB) IRAM 0/252kB(lfb 252kB) CPU [81%@1224,80%@1479,86%@1479,85%@1479] EMC_FREQ 3%@1600 GR3D_FREQ 0%@76 VIC_FREQ 0%@192 APE 25 PLL@27.5C CPU@31C PMIC@100C GPU@27C AO@39C thermal@32C POM_5V_IN 2628/3120 POM_5V_GPU 78/61 POM_5V_CPU 431/1086

After this I receive message on terminal kiled.

How Can I solve this issue ?

virajhapaliya · July 15, 2021, 9:31am

I was going through the NVIDIA developer Blog where they have mentioned that we can use this pose estimation in jetson or NVIDIA GPU.
But while loading the model if is crashing due the out of memory how can I test it in jetson nano.

AastaLLL · July 19, 2021, 7:31am

Hi,

Based on the memory usage from tegrastats, Nano is running out of memory when inferencing.

RAM 3888/3964MB ...
RAM 857/3964MB ...

We try to run the pose_estimation.onnx included in the GitHub with trtexec on Nano.
It can work correctly without causing the OOM issue.

Could you run the example with the pre-built pose_estimation.onnx first?

github.com

NVIDIA-AI-IOT/deepstream_pose_estimation/blob/master/pose_estimation.onnx

pytorch1.3:ÃÊÏ(

input.1
0.resnet.conv1.weight173"Conv*
	dilations@@ *
group *
kernel_shape@@ *
pads@@@@ *
strides@@ 
¢
173
0.resnet.bn1.weight
0.resnet.bn1.bias
0.resnet.bn1.running_mean
0.resnet.bn1.running_var174"BatchNormalization*
epsilon¬Å'7 *
momentumfff? 

174175"Relu
O

This file has been truncated. show original

Thanks.

virajhapaliya · July 19, 2021, 8:42am

I have run the original pose_estimation.onnx file from github with trtexec. I have attached the log here.

with_saveengine.txt : In this file I have run wihtout --saveEngine flag.
without_saveengine.txt :This contains log with --saveEngine flag.

with_saveengine.txt (3.7 KB)
without_saveengine.txt (4.8 KB)

AastaLLL · July 26, 2021, 6:59am

Hi,

Could you try to serialize the engine file with root authority?
Or another location doesn’t require root for writing?

And then please copy to engine file into the deepstream_pose_estimation sample and try it again.
Thanks.

virajhapaliya · July 26, 2021, 7:42am

Hi,

I have copied the pose_Estimation.onnx file to another folder where it doesn’t require root access. and I am still getting the same error in the with --saveEngine flag.

It is not able to save the engine file in the output folder.

AastaLLL · August 2, 2021, 6:10am

Hi,

Could you try a command like below?

$ /usr/src/tensorrt/bin/trtexec --onnx=pose_estimation.onnx --fp16 saveEngine=/home/nvidia/pose_estimation.onnx_b1_gpu0_fp16.engine
$ sudo cp /home/nvidia/pose_estimation.onnx_b1_gpu0_fp16.engine /opt/nvidia/deepstream/deepstream-5.1/sources/apps/sample_apps/deepstream_pose_estimation/out/

Thanks.

virajhapaliya · August 2, 2021, 8:58am

Hii,

I have executed the Command that you have posted here. It worked and I was able to generate the .engine file.

After that, I have run the command of pose estimation In the terminal but after loading the engine file Terminal was not printing any logs. I have waited for more than 25 minutes.

The video length is only 16 seconds. And command is in this log txt file

Logfile_pose_estimation.txt (1.6 KB)

Thank you

AastaLLL · August 9, 2021, 5:30am

Hi,

Have you maximized the Nano performance first?

$ sudo nvpmodel -m 0
$ sudo jetson_clocks

More, could you also run the tegrastats at the same time?

Since there is some swap memory used in the previous testing.
Swap is physically stored on the disk, it may cause some performance degradation.

Thanks.

virajhapaliya · August 9, 2021, 6:29am

Yes, I have tested this. And the Same Issue again.

RAM 2513/3964MB (lfb 107x4MB) SWAP 26/1982MB (cached 2MB) CPU [47%@1479,40%@1479,40%@1479,44%@1479] EMC_FREQ 0% GR3D_FREQ 0% PLL@29C CPU@32.5C PMIC@100C GPU@29.5C AO@40.5C thermal@31C
RAM 2514/3964MB (lfb 107x4MB) SWAP 26/1982MB (cached 2MB) CPU [35%@1479,25%@1479,26%@1479,34%@1479] EMC_FREQ 0% GR3D_FREQ 0% PLL@29.5C CPU@33C PMIC@100C GPU@29C AO@40C thermal@31C
RAM 2513/3964MB (lfb 107x4MB) SWAP 26/1982MB (cached 2MB) CPU [39%@1479,38%@1479,34%@1479,39%@1479] EMC_FREQ 0% GR3D_FREQ 4% PLL@29.5C CPU@33C PMIC@100C GPU@29.5C AO@40C thermal@31C
RAM 2513/3964MB (lfb 107x4MB) SWAP 26/1982MB (cached 2MB) CPU [34%@1479,30%@1479,27%@1479,34%@1479] EMC_FREQ 0% GR3D_FREQ 5% PLL@29C CPU@32.5C PMIC@100C GPU@29.5C AO@40C thermal@31C
RAM 2513/3964MB (lfb 107x4MB) SWAP 26/1982MB (cached 2MB) CPU [36%@1479,29%@1479,34%@1479,30%@1479] EMC_FREQ 0% GR3D_FREQ 3% PLL@29C CPU@32.5C PMIC@100C GPU@29C AO@40C thermal@30.75C
RAM 2512/3964MB (lfb 107x4MB) SWAP 26/1982MB (cached 2MB) CPU [29%@1479,25%@1479,23%@1479,27%@1479] EMC_FREQ 0% GR3D_FREQ 0% PLL@29C CPU@33C PMIC@100C GPU@29C AO@40C thermal@31C

These are the last logs.

Thanks

AastaLLL · August 16, 2021, 7:08am

Hi,

We want to reproduce this issue internally for further checking.
Would you mind sharing your testing video with us?

Thanks.

virajhapaliya · August 16, 2021, 1:24pm

Hi,

Please check the link below. In this link, there are two videos.

Link : Videos

Thanks

AastaLLL · August 23, 2021, 6:32am

Hi,

Thanks for sharing the video with us.
Would you mind enabling access to the video so we can download it?

Thanks.

virajhapaliya · August 23, 2021, 8:07am

Hii,

can you please request access? So I can give you permission.

Thanks.

AastaLLL · August 25, 2021, 6:12am

Sure. I have submitted the request.

virajhapaliya · August 25, 2021, 6:16am

Hello,

I have accepted the Request.

AastaLLL · August 25, 2021, 6:17am

Thanks.
Will share more informaiton with you later.

AastaLLL · September 1, 2021, 6:05am

Hi,

We can reproduce this issue in our environment.
It seems that the pipeline get stuck somehow.

Will share more information once we got a progress.
Thanks.