Runtime errors when running the human pose estimation application

please try H264?

please refer to code - deepstream_pose_estimation/deepstream_pose_estimation_app.cpp at master · NVIDIA-AI-IOT/deepstream_pose_estimation · GitHub , it only support H264 raw media file

1 Like

@mchi I tried running the program with the raw H264 media file. I still get an error. See the output below:

root@ea230df7fe23:/opt/nvidia/deepstream/deepstream-5.0/deepstream-pose-estimati
on# ./deepstream-pose-estimation-app stream.264 .
Now playing: stream.264
ERROR: ../nvdsinfer/nvdsinfer_model_builder.cpp:1523 Deserialize engine failed because file path: /opt/nvidia/deepstream/deepstream-5.0/deepstream-pose-estimation/pose_estimation.onnx_b1_gpu0_fp16.engine open error
0:00:00.712641747   180 0x55f78ba74790 WARN                 nvinfer gstnvinfer.cpp:616:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1690> [UID = 1]: deserialize engine from file :/opt/nvidia/deepstream/deepstream-5.0/deepstream-pose-estimation/pose_estimation.onnx_b1_gpu0_fp16.engine failed
0:00:00.712681947   180 0x55f78ba74790 WARN                 nvinfer gstnvinfer.cpp:616:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1797> [UID = 1]: deserialize backend context from engine from file :/opt/nvidia/deepstream/deepstream-5.0/deepstream-pose-estimation/pose_estimation.onnx_b1_gpu0_fp16.engine failed, try rebuild
0:00:00.712698347   180 0x55f78ba74790 INFO                 nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1715> [UID = 1]: Trying to create engine from model files
----------------------------------------------------------------
Input filename:   /opt/nvidia/deepstream/deepstream-5.0/deepstream-pose-estimation/resnet18_baseline_att_224x224_A_epoch_249.onnx
ONNX IR version:  0.0.6
Opset version:    9
Producer name:    pytorch
Producer version: 1.7
Domain:
Model version:    0
Doc string:
----------------------------------------------------------------
WARNING: ../nvdsinfer/nvdsinfer_model_builder.cpp:1291 FP16 not supported by platform. Using FP32 mode.
INFO: ../nvdsinfer/nvdsinfer_func_utils.cpp:39 [TRT]: Detected 1 inputs and 3 output network tensors.
0:00:35.641783401   180 0x55f78ba74790 INFO                 nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1748> [UID = 1]: serialize cuda engine to file: /opt/nvidia/deepstream/deepstream-5.0/deepstream-pose-estimation/resnet18_baseline_att_224x224_A_epoch_249.onnx_b1_gpu0_fp32.engine successfully
INFO: ../nvdsinfer/nvdsinfer_model_builder.cpp:685 [FullDims Engine Info]: layers num: 4
0   INPUT  kFLOAT input           3x224x224       min: 1x3x224x224     opt: 1x3x224x224     Max: 1x3x224x224
1   OUTPUT kFLOAT part_affinity_fields 56x56x42        min: 0               opt: 0               Max: 0
2   OUTPUT kFLOAT heatmap         56x56x18        min: 0               opt: 0               Max: 0
3   OUTPUT kFLOAT maxpool_heatmap 56x56x18        min: 0               opt: 0               Max: 0

0:00:35.653254679   180 0x55f78ba74790 INFO                 nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<primary-nvinference-engine> [UID 1]: Load new model:deepstream_pose_estimation_config.txt sucessfully
Running...
libnvosd (603):(ERROR) : Out of bound radius
ERROR from element nv-onscreendisplay: Unable to draw circles
Error details: gstnvdsosd.c(558): gst_nvds_osd_transform_ip (): /GstPipeline:deepstream-tensorrt-openpose-pipeline/GstNvDsOsd:nv-onscreendisplay
0:00:35.836519020   180 0x55f78bab8630 WARN                 nvinfer gstnvinfer.cpp:1975:gst_nvinfer_output_loop:<primary-nvinference-engine> error: Internal data stream error.
Returned, stopping playback
0:00:35.836566820   180 0x55f78bab8630 WARN                 nvinfer gstnvinfer.cpp:1975:gst_nvinfer_output_loop:<primary-nvinference-engine> error: streaming stopped, reason error (-5)
libnvosd (603):(ERROR) : Out of bound radius
libnvosd (603):(ERROR) : Out of bound radius
libnvosd (603):(ERROR) : Out of bound radius
libnvosd (603):(ERROR) : Out of bound radius
libnvosd (603):(ERROR) : Out of bound radius
Deleting pipeline

Could you tell me what is going wrong? I found this thread when looking for the error. Is this a bug with DeepStream itself? I’m running all of this via Putty (SSH) on a Microsoft Azure virtual machine if that’s relevant.

DId you follow the README in the github?

1 Like

Hello @mchi,

I forgot to replace the OSD binaries in the DeepStream Docker container. I did it now. Here is the output I get now:

root@723375a233fe:/opt/nvidia/deepstream/deepstream-5.0/deepstream-pose-estimati
on# ./deepstream-pose-estimation-app stream.264 .

(deepstream-pose-estimation-app:114): GStreamer-WARNING **: 07:28:04.437: Failed to load plugin '/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_osd.so': libcudart.so.11.0: cannot open shared object file: No such file or directory
One element could not be created. Exiting.

When I run gst-inspect-1.0 nveglglessink, it gives me the below output, which is telling me that there are no problems with the DeepStream SDK setup in my Docker container:

root@723375a233fe:/opt/nvidia/deepstream/deepstream-5.0/deepstream-pose-estimati
on# gst-inspect-1.0 nveglglessink
Factory Details:
  Rank                     secondary (128)
  Long-name                EGL/GLES vout Sink
  Klass                    Sink/Video
  Description              An EGL/GLES Video Output Sink Implementing the VideoOverlay interface
  Author                   Reynaldo H. Verdejo Pinochet <reynaldo@collabora.com>, Sebastian Dröge <sebastian.droege@collabora.co.uk>

Plugin Details:
  Name                     nvdsgst_eglglessink
  Description              EGL/GLES sink
  Filename                 /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_eglglessink.so
  Version                  5.0.1
  License                  LGPL
  Source module            gst-plugins-bad
  Source release date      2014-02-08
  Binary package           GStreamer Bad Plug-ins source release
  Origin URL               Unknown package origin

GObject
 +----GInitiallyUnowned
       +----GstObject
             +----GstElement
                   +----GstBaseSink
                         +----GstVideoSink
                               +----GstEglGlesSink

Implemented Interfaces:
  GstVideoOverlay

Pad Templates:
  SINK template: 'sink'
    Availability: Always
    Capabilities:
      video/x-raw(memory:EGLImage)
                 format: { (string)RGBA, (string)BGRA, (string)ARGB, (string)ABGR, (string)RGBx, (string)BGRx, (string)xRGB, (string)xBGR, (string)AYUV, (string)Y444, (string)I420, (string)YV12, (string)NV12, (string)NV21, (string)Y42B, (string)Y41B, (string)RGB, (string)BGR, (string)RGB16 }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
      video/x-raw(meta:GstVideoGLTextureUploadMeta)
                 format: { (string)RGBA, (string)BGRA, (string)ARGB, (string)ABGR, (string)RGBx, (string)BGRx, (string)xRGB, (string)xBGR, (string)AYUV, (string)Y444, (string)I420, (string)YV12, (string)NV12, (string)NV21, (string)Y42B, (string)Y41B, (string)RGB, (string)BGR, (string)RGB16 }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
      video/x-raw
                 format: { (string)RGBA, (string)BGRA, (string)ARGB, (string)ABGR, (string)RGBx, (string)BGRx, (string)xRGB, (string)xBGR, (string)AYUV, (string)Y444, (string)I420, (string)YV12, (string)NV12, (string)NV21, (string)Y42B, (string)Y41B, (string)RGB, (string)BGR, (string)RGB16 }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
      video/x-raw(memory:NVMM)
                 format: { (string)BGRx, (string)RGBA, (string)I420, (string)NV12, (string)BGR, (string)RGB }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]

Element has no clocking capabilities.
Element has no URI handling capabilities.

Pads:
  SINK: 'sink'
    Pad Template: 'sink'

Element Properties:
  name                : The name of the object
                        flags: readable, writable
                        String. Default: "eglglessink0"
  parent              : The parent of the object
                        flags: readable, writable
                        Object of type "GstObject"
  sync                : Sync on the clock
                        flags: readable, writable
                        Boolean. Default: true
  max-lateness        : Maximum number of nanoseconds that a buffer can be late before it is dropped (-1 unlimited)
                        flags: readable, writable
                        Integer64. Range: -1 - 9223372036854775807 Default: 20000000
  qos                 : Generate Quality-of-Service events upstream
                        flags: readable, writable
                        Boolean. Default: true
  async               : Go asynchronously to PAUSED
                        flags: readable, writable
                        Boolean. Default: true
  ts-offset           : Timestamp offset in nanoseconds
                        flags: readable, writable
                        Integer64. Range: -9223372036854775808 - 9223372036854775807 Default: 0
  enable-last-sample  : Enable the last-sample property
                        flags: readable, writable
                        Boolean. Default: true
  last-sample         : The last sample received in the sink
                        flags: readable
                        Boxed pointer of type "GstSample"
  blocksize           : Size in bytes to pull per buffer (0 = default)
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 4294967295 Default: 4096
  render-delay        : Additional render delay of the sink in nanoseconds
                        flags: readable, writable
                        Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0
  throttle-time       : The time to keep between rendered buffers (0 = disabled)
                        flags: readable, writable
                        Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0
  max-bitrate         : The maximum bits per second to render (0 = disabled)
                        flags: readable, writable
                        Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0
  show-preroll-frame  : Whether to render video frames during preroll
                        flags: readable, writable
                        Boolean. Default: true
  create-window       : If set to true, the sink will attempt to create it's own window to render to if none is provided. This is currently only supported when the sink is used under X11
                        flags: readable, writable
                        Boolean. Default: true
  force-aspect-ratio  : If set to true, the sink will attempt to preserve the incoming frame's geometry while scaling, taking both the storage's and display's pixel aspect ratio into account
                        flags: readable, writable
                        Boolean. Default: true
  display             : If set, the sink will use the passed X Display for rendering
                        flags: readable, writable
                        Pointer.
  window-x            : X coordinate of window
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 2147483647 Default: 10
  window-y            : Y coordinate of window
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 2147483647 Default: 10
  window-width        : Width of window
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 2147483647 Default: 0
  window-height       : Height of window
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 2147483647 Default: 0
  rows                : Rows of Display
                        flags: readable, writable
                        Unsigned Integer. Range: 1 - 2147483647 Default: 1
  columns             : Columns of display
                        flags: readable, writable
                        Unsigned Integer. Range: 1 - 2147483647 Default: 1
  gpu-id              : Set GPU Device ID
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Integer. Range: 0 - 4294967295 Default: 0
  profile             : gsteglglessink jitter information
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 4294967295 Default: 0

I have to note that I copied the ONNX model and the compiled program that I already had to the Docker container. I don’t think this affects the process, I’m just noting it here if it’s relevant.

What do you think the issue is now?

I noticed that in my docker container, when I try to nvcc --version, I get:

root@64380ac508a2:/opt/nvidia/deepstream/deepstream-5.0# nvcc --version
bash: nvcc: command not found

I think this may be why I’m getting the error. I am using the base tag from the DeepStream Docker containers. Should I use the devel tag DeepStream Docker container for running the pose estimation app?

yes, you’d better to use DS NGC docker, but I found the osd library is based on CUDA 11, which should be not correct for current DS since current DS is based on CUDA 10.2.

So, I would recommend to use NGC DS docker, and don’t replace the osd lib for now.
To avoid the OSD error you met above, you could keep the all OSD in image, that is, no OSD draw outside of the image,

So if I understood you correctly:

  • I should use DeepStream Docker container with the devel tag from here
  • I don’t have to re-convert the model to ONXX or re-compile the program
  • I should draw all the on-screen drawings in an image to avoid the error Failed to load plugin '/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_osd.so': libcudart.so.11.0: cannot open shared object file: No such file or directory

How would I do the 3rd point, that is, how should I draw all the on-screen drawings in an image?

Hey @mchi,

I followed the tutorial from scratch again today. I used the PyTorch container for converting the model to the ONXX format and for everything else I used the DeepStream Docker container with the tag devel. In other words, I built the app in the DeepStream Docker container. I made sure I followed all of the instructions.

This is the output that I get:

root@4f541fd78b91:/opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps
/deepstream_pose_estimation# ./deepstream-pose-estimation-app stream.264 .

(gst-plugin-scanner:46): GStreamer-WARNING **: 11:43:50.268: Failed to load plugin '/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_inferserver.so': libtrtserver.so: cannot open shared object file: No such file or directory

(gst-plugin-scanner:46): GStreamer-WARNING **: 11:43:50.272: Failed to load plugin '/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_osd.so': libcudart.so.11.0: cannot open shared object file: No such file or directory
One element could not be created. Exiting.

Note that there’s two warnings now, not just one. And this is from trying to run the app in the same container that I made it in, that is, the DeepStream Docker container with the tag devel. Do you know what’s happening here? How do I fix this?

We need go back to Post#10 - Runtime errors when running the human pose estimation application - #10 by mjuric1 , where you run into the OSD error - “libnvosd (603):(ERROR) : Out of bound radius”. This is a DeepStream OSD bug which the OSD lib libnvds_osd.so targets to fix, but this lib is not correct since it’s based on CUDA 11, so for now., to avoid the OSD error , could use below temporary change.
NVIDIA is in holiday this week, we will update the OSD lib after holiday.

--- a/deepstream_pose_estimation_app.cpp
+++ b/deepstream_pose_estimation_app.cpp
@@ -95,6 +95,14 @@ create_display_meta(Vec2D<int> &objects, Vec3D<float> &normalized_peaks, NvDsFra
         auto &peak = normalized_peaks[j][k];
         int x = peak[1] * MUXER_OUTPUT_WIDTH;
         int y = peak[0] * MUXER_OUTPUT_HEIGHT;
+       if ((x > (1920 - 8)) || (y > (1080 - 8))) {
+                printf("x = %d, y = %d\n", x, y);
+                continue;
+        }
+       if ((x < 8) || (y < 8)) {
+               printf("x = %d, y = %d\n", x, y);
+                continue;
+       }
         if (dmeta->num_circles == MAX_ELEMENTS_IN_DISPLAY_META)
         {
1 Like

@mchi This works in the sense that the program is able to run and produce output. Thank you. The quality of the output (pose estimation in the video) is another story…

@wade.wang @mchi provided a solution here. Instead of replacing the DeepStream OSD library, add this code in deepstream_pose_estimation_app.cpp instead. Also, make sure to input raw H264 stream; to see how to do that, look here: matlab - How to extract the bitstream from H.264 video? - Stack Overflow

1 Like

@mjuric1 @mchi Hi,there, great work! Very happy to hear that the solution has been found, in my case, this solution also works, thank you very much !

@wade.wang I wanted to ask you one thing - do the results that the program outputs predict the pose correctly for you? In my case, the program outputs some nonsensical points and doesn’t estimate the pose at all.

Hi, @mjuric1, please look at video of my test results: link.
It seems the result is not very good for small and occluded person.

1 Like

Hey @wade.wang, thank you for responding.If you don’t mind, could you send me the original video (preferably already in raw H264 format) so that I can try it on my machine? The results I get are far, far worse… I don’t even get the green connected lines, I get a random set of points which aren’t connected.

I also noticed that my input video gets rotated by 90 degrees to the left when the pose estimation application outputs it.

Also, could you provide me with the command with which you run the pose estimation app (something like ./deepstream-pose-estimation-app input.264 <outputPath>, but your actual command?). I’m trying to figure out what’s going wrong on my end.

One additional question: Do you use ResNet or DenseNet?

Sure, this is the orginal mp4 video, and this the converted H264 file.

My command line used is sudo ./deepstream-pose-estimation-app /media/weidawang/DATA/dataset/stream.264 output-results

No, i don’t even know what is the usage of ResNet or DenseNet in deepstream_pose_estimation, i didn’t do any extra actions other than those mentioned in that blog.

1 Like

@wade.wang thank you for providing the input files.

I tried your input files on my machine and the results are nowhere near what you’re getting. I’m getting some random points on my screen and no green lines whatsoever. I’ll open up a new post with exactly the steps to re-create my problem so that people can tell me what I’m doing wrong.

Thank you for helping out. I’ll tag you in the new post as well, so maybe you could take a look at the process with which I set up my project. Probably something is going wrong there.

An update: I figured out what the problem was. It is related to the conversion of the weights to the ONNX format. I left the default weights (pose_esimation.onnx) in the deepstream_pose_estimation_config.txt file and I get the same results as you do.

Thank you for all your responses.

Congratulation ! I also used the pose_esimation.onnx which is cloned with the repo in the root directory.

1 Like