Creating a Human Pose Estimation Application with NVIDIA DeepStream

Hi,
Please check deepstream_pose_estimation/deepstream_pose_estimation_app.cpp at 4f79d2d1cf65e3c992c261c90b720a4c69230fdd · NVIDIA-AI-IOT/deepstream_pose_estimation · GitHub to see how the data is manipulated after detection.
Hope this answers your question!

Hi,
You probably want to update the window size in post processing to find the peaks.

Not fully really. The question is in simple terms, which is the array containing body parts and coordinates? And how do I know which element corresponds to each body part? I can’t find labels anywhere.

thanks a lot

Thanks, but not this problem actually, i check Integrated deepstream-pose-estimation into deepstream-app with smalll display issue and find my problem is the same. So does DS5.1 can help solving it?

I am also trying to determine coordinates for specific body parts. Have you had any progress with this?

Hi, I met this error, what should I do?

root@nx-desktop:/opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/deepstream_pose_estimation# ./deepstream-pose-estimation-app sample_qHD.h264   ./
Now playing: sample_qHD.h264
Opening in BLOCKING MODE 
Opening in BLOCKING MODE 
0:00:04.068806941 14643   0x5571807c00 INFO                 nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1701> [UID = 1]: deserialized trt engine from :/opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/deepstream_pose_estimation/resnet18_baseline_att_224x224_A_epoch_249.onnx_b1_gpu0_fp16.engine
INFO: [Implicit Engine Info]: layers num: 3
0   INPUT  kFLOAT input           3x224x224       
1   OUTPUT kFLOAT 262             18x56x56        
2   OUTPUT kFLOAT 264             42x56x56        

0:00:04.069061365 14643   0x5571807c00 INFO                 nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1805> [UID = 1]: Use deserialized engine model: /opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/deepstream_pose_estimation/resnet18_baseline_att_224x224_A_epoch_249.onnx_b1_gpu0_fp16.engine
0:00:04.074797107 14643   0x5571807c00 INFO                 nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<primary-nvinference-engine> [UID 1]: Load new model:deepstream_pose_estimation_config.txt sucessfully
Running...
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
NvMMLiteOpen : Block : BlockType = 4 
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
H264: Profile = 66, Level = 0 
End of Stream
Returned, stopping playback
Deleting pipeline

I worked out a way to access the coordinates. Look at the objects and peaks tensors. The objects tensor can be converted into a 1x100x18 numpy array. Each row is a what the model believes is an individual, so if you’re only observing one person, you should only have to look at the first row. Each of the 18 elements in the row map to the body part keypoints in the human_pose JSON COCO object. The objects array only shows things the model identified. To get coordinates, look at the peaks tensor.

Convert peaks to a numpy array. it will be 1x18x100x2. The 18 axis correlates to the 18 keypoints in the objects array. The 100 correlates to the 100 total possible individuals the model can simultaneously track. The 2 axis is your x & y coordinate, normalized to [0-1].

For instance, I was looking for “right_elbow” and “right_wrist” - keypoints 8 & 10. When I send an image to the model, the 0 row of the objects tensor/array shows [-1,-1,-1, -1,-1,-1,-1,-1,0,-1,0,-1,-1,-1,-1,-1,-1,-1]. The 0’s mark the keypoints found in my image (right wrist and elbow).

Next, look into the peaks tensor at element. For my example, it is observation 0, keypoints 8 & 10. In my example, I converted peaks to a numpy array named coords. I accessed what I needed by:
x1 = coords[0][8][0][1]
y1 = coords[0][8][0][0]

x2 = coords[0][10][0][1]
y2 = coords[0][10][0][0]

I outputted these values to the screen, and on a live feed, got the attached screen cap.

image

I hope that helps. If some code example might help, I can share it, too.

1 Like

Thank you @bradpr! This is so useful.

1 Like

The coordinate system used is inverted. 0,0 is top right, so you will have to transform to get a normal X,Y origin at the bottom left.

1 Like

Evaluating the objects tensor, I have values -1, 0 and 1, I guess that -1 is absence of detection and 0 is presense. Anyone knows how to interpret the value “1”? Thanks

you are the man

1 Like

Hi,
I run your app successfully. But I want use your app for other model. I just change the post-process code to parse pose from output but when I run my custom app. It raise this error:

ERROR from element h264-parser: Internal data stream error.
Error details: gstbaseparse.c(3611): gst_base_parse_loop (): /GstPipeline:deepstream-tensorrt-openpose-pipeline/GstH264Parse:h264-parser:
streaming stopped, reason not-negotiated (-4)

I check my video file, it is h264 format. Can you help me to fix it? Thank you very much! My deepstream app file

Hi Tucachmo,

Could you please explain what all updates you made in the source code?

Thanks for replying. I just change the engine file and rewrite parse keypoints and poses from output model. However, I found that in previous project I replace some default lib so it caused error.

Hi,

I was wondering if the model could be used a secondary gie operating on primary detections from something like yolo. Has anyone done this ?

Thanks a million

Hi,

I have a nano 4G, I have been trying to get the pose estimation working for a month, no luck. I found your blog, but there are some modification I need to do, may I know if anyone can sucessfully get the pose estimation working?

nvidia@nvidia-desktop:/opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/deepstream_pose_estimation$ sudo make
g++ -c -o deepstream_pose_estimation_app.o -DPLATFORM_TEGRA -I../../apps-common/includes -I../../../includes -I../deepstream-app/ -DDS_VERSION_MINOR=0 -DDS_VERSION_MAJOR=5 -pthread -I/usr/include/gstreamer-1.0 -I/usr/include/orc-0.4 -I/usr/include/gstreamer-1.0 -I/usr/include/json-glib-1.0 -I/usr/include/glib-2.0 -I/usr/lib/aarch64-linux-gnu/glib-2.0/include deepstream_pose_estimation_app.cpp
deepstream_pose_estimation_app.cpp: In function ‘GstPadProbeReturn osd_sink_pad_buffer_probe(GstPad*, GstPadProbeInfo*, gpointer)’:
deepstream_pose_estimation_app.cpp:231:77: warning: zero-length gnu_printf format string [-Wformat-zero-length]
     offset = snprintf(txt_params->display_text + offset, MAX_DISPLAY_LEN, "");
                                                                             ^
deepstream_pose_estimation_app.cpp:236:41: warning: ISO C++ forbids converting a string constant to ‘char*’ [-Wwrite-strings]
     txt_params->font_params.font_name = "Mono";
                                         ^~~~~~
g++ -o deepstream-pose-estimation-app deepstream_pose_estimation_app.o -L/opt/nvidia/deepstream/deepstream-5.0/lib/ -lnvdsgst_meta -lnvds_meta -lnvds_utils -lm -lpthread -ldl -Wl,-rpath,/opt/nvidia/deepstream/deepstream-5.0/lib/ -lgstvideo-1.0 -lgstbase-1.0 -lgstreamer-1.0 -lX11 -ljson-glib-1.0 -lgio-2.0 -lgobject-2.0 -lglib-2.0
nvidia@nvidia-desktop:/opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/deepstream_pose_estimation$ sudo ./deepstream-pose-estimation-app test.mp4 .
One element could not be created. Exiting.
nvidia@nvidia-desktop:/opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/deepstream_pose_estimation$ sudo ./deepstream-pose-estimation-app test.mp4 Usage: ./deepstream-pose-estimation-app <filename> <output-path>
nvidia@nvidia-desktop:/opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/deepstream_pose_estimation$ sudo ./deepstream-pose-estimation-app test.mp4 .
One element could not be created. Exiting.
nvidia@nvidia-desktop:/opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/deepstream_pose_estimation$ 

Here are what I did

Following this link
https://github.com/NVIDIA-AI-IOT/deepstream_pose_estimation#:~:text=Human%20pose%20estimation%20is%20the,a%20video%20or%20a%20photo.

1. Install DeepStream on your platform, verify it is working by running deepstream-app.
Download DeepStream 5.0to my Downloads folder
https://developer.nvidia.com/assets/Deepstream/5.0/ga/secure/deepstream_sdk_5.0_jetson.tbz2

Follow the setup guide "Install Jetson SDK compenents"
https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_Quickstart.html#jetson-setup

nvidia@nvidia-desktop:~$ sudo apt install \
> libssl1.0.0 \
> libgstreamer1.0-0 \
> gstreamer1.0-tools \
> gstreamer1.0-plugins-good \
> gstreamer1.0-plugins-bad \
> gstreamer1.0-plugins-ugly \
> gstreamer1.0-libav \
> libgstrtspserver-1.0-0 \
> libjansson4=2.11-1
Reading package lists... Done
Building dependency tree       
Reading state information... Done
E: Unable to locate package libgstrtspserver-1.0-0
E: Couldn't find any package by glob 'libgstrtspserver-1.0-0'
E: Couldn't find any package by regex 'libgstrtspserver-1.0-0'
nvidia@nvidia-desktop:~$ 

	Follow this link, https://zoomadmin.com/HowToInstall/UbuntuPackage/libgstrtspserver-1.0-0
	sudo apt-get update -y
	sudo apt-get install -y libgstrtspserver-1.0-0

nvidia@nvidia-desktop:~$ sudo apt install \
> libssl1.0.0 \
> libgstreamer1.0-0 \
> gstreamer1.0-tools \
> gstreamer1.0-plugins-good \
> gstreamer1.0-plugins-bad \
> gstreamer1.0-plugins-ugly \
> gstreamer1.0-libav \
> libgstrtspserver-1.0-0 \
> libjansson4=2.11-1
git clone https://github.com/edenhill/librdkafka.git
cd librdkafka
git reset --hard 7101c2310341ab3f4675fc565f64f0967e135a6a
./configure
make
sudo make install
#need to modify for 5.0
sudo mkdir -p /opt/nvidia/deepstream/deepstream-5.0/lib
sudo cp /usr/local/lib/librdkafka* /opt/nvidia/deepstream/deepstream-5.0/lib

sudo vi /etc/apt/sources.list.d/nvidia-l4t-apt-source.list
Modify the code to 
	deb https://repo.download.nvidia.com/jetson/common r32.6 main
	deb https://repo.download.nvidia.com/jetson/t194 r32.6 main

sudo apt update
sudo apt install --reinstall nvidia-l4t-gstreamer
sudo apt install --reinstall nvidia-l4t-multimedia
sudo apt install --reinstall nvidia-l4t-core
cd
cd Downloads
sudo tar -xvf deepstream_sdk_5.0_jetson.tbz2 -C /
cd /opt/nvidia/deepstream/deepstream-5.0
sudo ./install.sh
sudo ldconfig
cd /opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps
sudo git clone https://github.com/NVIDIA-AI-IOT/deepstream_pose_estimation.git
cd
#########################################
#Goto trtpose
#Install pytorch and torchyvision
#Follow: https://forums.developer.nvidia.com/t/pytorch-for-jetson-version-1-10-now-available/72048
wget https://nvidia.box.com/shared/static/p57jwntv436lfrd78inwl7iml6p13fzh.whl -O torch-1.8.0-cp36-cp36m-linux_aarch64.whl
sudo apt-get install python3-pip libopenblas-base libopenmpi-dev 
pip3 install Cython
pip3 install numpy torch-1.8.0-cp36-cp36m-linux_aarch64.whl
##
##torchvision
sudo apt-get install libjpeg-dev zlib1g-dev libpython3-dev libavcodec-dev libavformat-dev libswscale-dev
git clone --branch v0.9.0 https://github.com/pytorch/vision torchvision
cd torchvision
export BUILD_VERSION=0.9.0
python3 setup.py install --user
cd ../
pip install 'pillow<7'
#####################################
# pip not found
apt-get install python-pip
##################################
pip install 'pillow<7'

##########################################################
git clone https://github.com/NVIDIA-AI-IOT/torch2trt
cd torch2trt
sudo python3 setup.py install --plugins
#########################################################

###################################################
#ModuleNotFoundError: No module named 'packaging'
#from this link, https://stackoverflow.com/questions/42222096/no-module-named-packaging
pip3 install --upgrade pip
pip3 install packaging
#####################################################

sudo python3 setup.py install --plugins

sudo pip3 install tqdm cython pycocotools
sudo apt-get install python3-matplotlib


cd
git clone https://github.com/NVIDIA-AI-IOT/trt_pose
cd trt_pose
sudo python3 setup.py install

#Download the two model
https://drive.google.com/file/d/1XYDdCUdiF2xxx4rznmLb62SdOUZuoNbd/view?usp=drive_open
https://drive.google.com/file/d/13FkJkx7evQ1WwP54UmdiDXWyFMY1OxDU/view?usp=drive_open
#copy these two models to Place the downloaded weights in the tasks/human_pose directory
#copy these two models to Place the downloaded weights in the ~trt_pose/trt_pose/utils directory
cd
cd trt_pose/trt_pose/utils
#error
# follow this link
https://forums.developer.nvidia.com/t/conversion-of-model-weights-for-human-pose-estimation-model-to-onnx-results-in-nonsensical-pose-estimation/164417/11
#freeze
#need to increase the swap memory
python3 export_for_isaac.py --input_checkpoint resnet18_baseline_att_224x224_A_epoch_249.pth --input_topology ../../tasks/human_pose/human_pose.json

cd
git clone https://github.com/NVIDIA-AI-IOT/deepstream_pose_estimation.git
#copy the onnx file to deepstream_pose_estimation
#Replace the OSD binaries (x86 or Jetson) in $DEEPSTREAM_DIR/libs with the ones provided in this #repository under bin/. Please note that these are not inter-compatible across platforms.

nvidia@nvidia-desktop:~/deepstream_pose_estimation/bin$ ls
Jetson  x86
nvidia@nvidia-desktop:~/deepstream_pose_estimation/bin$ cd Jetson/
nvidia@nvidia-desktop:~/deepstream_pose_estimation/bin/Jetson$ ls
libnvds_osd.so
nvidia@nvidia-desktop:~/deepstream_pose_estimation/bin/Jetson$ sudo cp libnvds_osd.so /opt/nvidia/deepstream/deepstream-5.0/lib
[sudo] password for nvidia: 
nvidia@nvidia-desktop:~/deepstream_pose_estimation/bin/Jetson$ 

nvidia@nvidia-desktop:~/deepstream_pose_estimation$ sudo make
Package json-glib-1.0 was not found in the pkg-config search path.
Perhaps you should add the directory containing `json-glib-1.0.pc'
to the PKG_CONFIG_PATH environment variable
No package 'json-glib-1.0' found
g++ -c -o deepstream_pose_estimation_app.o -DPLATFORM_TEGRA -I../../apps-common/includes -I../../../includes -I../deepstream-app/ -DDS_VERSION_MINOR=0 -DDS_VERSION_MAJOR=5  deepstream_pose_estimation_app.cpp
In file included from deepstream_pose_estimation_app.cpp:4:0:
post_process.cpp:8:10: fatal error: gst/gst.h: No such file or directory
 #include <gst/gst.h>
          ^~~~~~~~~~~
compilation terminated.
Makefile:44: recipe for target 'deepstream_pose_estimation_app.o' failed
make: *** [deepstream_pose_estimation_app.o] Error 1
nvidia@nvidia-desktop:~/deepstream_pose_estimation$ 
sudo apt-get install libgstreamer1.0-dev
sudo apt install libjson-glib-dev

nvidia@nvidia-desktop:~/deepstream_pose_estimation$ sudo make
g++ -c -o deepstream_pose_estimation_app.o -DPLATFORM_TEGRA -I../../apps-common/includes -I../../../includes -I../deepstream-app/ -DDS_VERSION_MINOR=0 -DDS_VERSION_MAJOR=5 -pthread -I/usr/include/gstreamer-1.0 -I/usr/include/orc-0.4 -I/usr/include/gstreamer-1.0 -I/usr/include/json-glib-1.0 -I/usr/include/glib-2.0 -I/usr/lib/aarch64-linux-gnu/glib-2.0/include deepstream_pose_estimation_app.cpp
In file included from deepstream_pose_estimation_app.cpp:4:0:
post_process.cpp:12:10: fatal error: gstnvdsmeta.h: No such file or directory
 #include "gstnvdsmeta.h"
          ^~~~~~~~~~~~~~~
compilation terminated.
Makefile:44: recipe for target 'deepstream_pose_estimation_app.o' failed
make: *** [deepstream_pose_estimation_app.o] Error 1

#copy all files in /opt/nvidia/deepstream/deepstream-5.1/sources/includes/ to deepstream_pose_estimation
sudo make
#download a crowd video from https://www.pexels.com/search/videos/crowd/
$copy the video.mp4 to :~/deepstream_pose_estimation

sudo ./deepstream-pose-estimation-app test.mp4 
nvidia@nvidia-desktop:~/deepstream_pose_estimation$ sudo ./deepstream-pose-estimation-app test.mp4 .
One element could not be created. Exiting.
nvidia@nvidia-desktop:~/deepstream_pose_estimation$ sudo ./deepstream-pose-estimation-app test.mp4 Pose_Estimation.mp4
One element could not be created. Exiting.
nvidia@nvidia-desktop:~/deepstream_pose_estimation$ 

#edit deepstream_pose_estimation_config.txt
[property]
gpu-id=0
net-scale-factor=0.0174292
offsets=123.675;116.28;103.53
onnx-file=resnet18_baseline_att_224x224_A_epoch_249.onnx
labelfile-path=labels.txt
batch-size=1
process-mode=1
model-color-format=0
## 0=FP32, 1=INT8, 2=FP16 mode
network-mode=2
num-detected-classes=4
interval=0
gie-unique-id=1
model-engine-file=pose_estimation.onnx_b1_gpu0_fp16.engine
network-type=100
workspace-size=3000


##################################################################
#I was doing the deepstream in another folder. :P
#I copy all files to the correct folder, touch the make and rebuild it again

nvidia@nvidia-desktop:/opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/deepstream_pose_estimation$ sudo make
g++ -c -o deepstream_pose_estimation_app.o -DPLATFORM_TEGRA -I../../apps-common/includes -I../../../includes -I../deepstream-app/ -DDS_VERSION_MINOR=0 -DDS_VERSION_MAJOR=5 -pthread -I/usr/include/gstreamer-1.0 -I/usr/include/orc-0.4 -I/usr/include/gstreamer-1.0 -I/usr/include/json-glib-1.0 -I/usr/include/glib-2.0 -I/usr/lib/aarch64-linux-gnu/glib-2.0/include deepstream_pose_estimation_app.cpp
deepstream_pose_estimation_app.cpp: In function ‘GstPadProbeReturn osd_sink_pad_buffer_probe(GstPad*, GstPadProbeInfo*, gpointer)’:
deepstream_pose_estimation_app.cpp:231:77: warning: zero-length gnu_printf format string [-Wformat-zero-length]
     offset = snprintf(txt_params->display_text + offset, MAX_DISPLAY_LEN, "");
                                                                             ^
deepstream_pose_estimation_app.cpp:236:41: warning: ISO C++ forbids converting a string constant to ‘char*’ [-Wwrite-strings]
     txt_params->font_params.font_name = "Mono";
                                         ^~~~~~
g++ -o deepstream-pose-estimation-app deepstream_pose_estimation_app.o -L/opt/nvidia/deepstream/deepstream-5.0/lib/ -lnvdsgst_meta -lnvds_meta -lnvds_utils -lm -lpthread -ldl -Wl,-rpath,/opt/nvidia/deepstream/deepstream-5.0/lib/ -lgstvideo-1.0 -lgstbase-1.0 -lgstreamer-1.0 -lX11 -ljson-glib-1.0 -lgio-2.0 -lgobject-2.0 -lglib-2.0
nvidia@nvidia-desktop:/opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/deepstream_pose_estimation$ sudo ./deepstream-pose-estimation-app test.mp4 .
One element could not be created. Exiting.
nvidia@nvidia-desktop:/opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/deepstream_pose_estimation$ sudo ./deepstream-pose-estimation-app test.mp4 Usage: ./deepstream-pose-estimation-app <filename> <output-path>
nvidia@nvidia-desktop:/opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/deepstream_pose_estimation$ sudo ./deepstream-pose-estimation-app test.mp4 .
One element could not be created. Exiting.
nvidia@nvidia-desktop:/opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/deepstream_pose_estimation$ 


















This will make your system upgrade to Jetpack4.6, and Jetpack4.6 is incompatiable with DeepStream 5.0 which package you downloaded.
So, I suggest below steps:

  1. flash the device again with Jetpack4.6 via SDKManager and install the all the Jetson SDK components.

  1. After flash & system boots up, install DeepStream Dependencies - Quickstart Guide — DeepStream 6.0 Release documentation (skip the steps - Install latest NVIDIA BSP packages)
  2. Install DeepStream6.0GA - Quickstart Guide — DeepStream 6.0 Release documentation
  3. Install and run deepstream_pose_estimation
1 Like

Hi,

Thx, I didn’t know jetpack 4.6 does not work well with deepstream5.0.
And I use the deepstream 6.0 instead with some tunning, the pose estimation works now. Thx
I found the solution in here

Hi!I have successfully run the whole project, and it is running well so far, but I found that there are some bone points that will be recognized elsewhere when they are recognized, such as I am sitting on a chair, my elbow is on the arm of the chair, and I am turning When the arm is used, the bone points of the wrist often identify the armrest of the chair. I noticed that there are several parameters in parse_objects_from_tensor_meta in deepstream_pose_estimation_app.cpp, but I don’t know the specific meaning of each parameter, can you help me Can you explain, thank you very much, in addition, the effect of this project is really good, looking forward to your reply!

Hi @user102939 ,
You may take a look [UPDATE 2/25/2021 to include hand pose] Real time human pose estimation on Jetson Nano (22FPS) - #15 by jaybdub to check if it could help you understand the post-processor.

1 Like