Testing pose-estimator in deepstream

[edit_or]

• Hardware Platform (Jetson / GPU)
xavier
• DeepStream Version
6.0
• JetPack Version (valid for Jetson only)
4.6
• TensorRT Version
TRT 8.0.1
• NVIDIA GPU Driver Version (valid for GPU only)
10.2

I am testing humanpose application.

I have already created onnx file for the given model here.
resnet18_baseline_att_224x224_A_epoch_249.onnx (79.2 MB)

The configuration file is

[property]
gpu-id=0
net-scale-factor=0.0174292
offsets=123.675;116.28;103.53
onnx-file=densenet121_baseline_att_256x256_B_epoch_160.onnx
labelfile-path=labels.txt
batch-size=1
process-mode=1
model-color-format=0
## 0=FP32, 1=INT8, 2=FP16 mode
network-mode=2
num-detected-classes=4
interval=0
gie-unique-id=1
model-engine-file=densenet121_baseline_att_256x256_B_epoch_160.onnx_b1_gpu0_fp16.engine
network-type=100
workspace-size=3000

When I run the app as
./deepstream-pose-estimation-app /opt/nvidia/deepstream/deepstream-6.0/samples/stream/sample_720p.h264 sample.mp4

Output mp4 has no detection. What is wrong with my test?
You can see the output video, there are circles plotted. But not at right place.

[edit_or]

Any response? Why it doesn’t work? Did I do sth wrong?

[Amycao]

Sorry for the late.
I will try to see and get back to you later.

[edit_or]

Thanks I have one project to use that app

[Amycao]

Any changes made in the post processing for the model? if yes, you need to change post processing code accordingly.

[edit_or]

I didn’t do anything. Just convert to onnx as mentioned and follow the instructions.

[Amycao]

The post processing in the github is for model output dimension format in CHW, but the model you provided is in HWC format. you need to change the post processing accordingly.

[edit_or]

I used this export_for_isaac.py provided in deepstream_pose_estimation. Model also downloaded from the mentioned link.. Which post processing part I need to update?
Is it possible to change in export_for_isaac.py?

[Amycao]

Supposed you used NVIDIA-AI-IOT/deepstream_pose_estimation: This is a sample DeepStream application to demonstrate a human pose estimation pipeline. (github.com), if you need this, you need to change the post processing accordingly.
There no export_for_isaac.py in this github, i think you should use another related github app? The model you attached is from another github NVIDIA-AI-IOT/trt_pose: Real-time pose estimation accelerated with NVIDIA TensorRT (github.com)

[edit_or]

No they mentioned in Getting Started place to use which model and export utility code. I just used the mentioned model and code.

[edit_or]

I changed in export program from 0,2,3,1 (nhwc) to 0,1,2,3 (nchw)

[Amycao]

I changed in export program from 0,2,3,1 (nhwc) to 0,1,2,3 (nchw)

Did it solve the issue?

[edit_or]

I see some detection on person for resnet model. But not accurate.
Densenet has no detection. I’ll upload detection tomorrow.
Current application works only for h264 video, not for other format.
So I update application to work for any videos format and test to see it really works or not.
I have only one h264 video, so it works or doesn’t work, still don’t know.

[edit_or]

The output tensor shapes are changed from nchw to nhwc. I checked using netron.

[edit_or]

Here is the output tensor shapes after changing from nhwc to nchw.

NHWC

NCHW

[mchi]

Hi @edit_or ,
By comparing resnet18_baseline_att_224x224_A_epoch_249.onnx with the pose_estimation.onnx in humanpose, you could find there are some differences on input and output between them as screenshots below.

So we need to do some revision to resnet18_baseline_att_224x224_A_epoch_249.onnx with script attached.
How to use this script (you could run this in a pytorch docker, e.g. nvcr.io/nvidia/pytorch:22.03-py3)
$ python3 -m pip install onnx_graphsurgeon --index-url https://pypi.ngc.nvidia.com
// only keep the output “onnx::MaxPool_266” and “onnx::Transpose_268”
$ python3 revise_model_for_deepstream_pose_github.py
revise_model_for_deepstream_pose_github.py (1.6 KB)
dynamic_resnet18_baseline_att_224x224_A_epoch_249.onnx “onnx::MaxPool_266” “onnx::Transpose_268”

and, then replace the pose_estimation.onnx in humanpose with this output onnx model .

Input

Output

[mchi]

And, I found your resnet18_baseline_att_224x224_A_epoch_249.onnx is fixed batch with batchsize==1, you could also apply below change to GitHub - NVIDIA-AI-IOT/trt_pose: Real-time pose estimation accelerated with NVIDIA TensorRT project to export dynamic batch onnx model

/**** Apply below change to export dynamic batch ****/

	root@b67263257cca:~/trt_pose/trt_pose/utils# git diff
	diff --git a/trt_pose/utils/export_for_isaac.py b/trt_pose/utils/export_for_isaac.py
	index f13bbc7..01d8c56 100644
	--- a/trt_pose/utils/export_for_isaac.py
	+++ b/trt_pose/utils/export_for_isaac.py
	@@ -127,8 +127,10 @@ def main(args):
	
	     # Export the model to ONNX.
	     dummy_input = torch.zeros((1, 3, args.input_height, args.input_width)).cuda()
	-    torch.onnx.export(converted_model, dummy_input, args.output_model,
	-                      input_names=input_names, output_names=output_names)
	+    dynamic_axes = {dummy_input : {0 : 'batch_size'}}
	+    torch.onnx.export(converted_model, dummy_input, args.output_model, opset_version=13, do_constant_folding=True,
	+                      input_names=input_names, output_names=output_names,
	+                      dynamic_axes={"input":{0:"batch_size"}, "output":{0:"batch_size"}})
	     print("Successfully completed convertion of %s to %s." % (args.input_checkpoint, args.output_model))

[edit_or]

Ok thanks let me try.

[edit_or]

Yes it works. I’ll proceed for dense model. Thank you