Run deepstream-bodypose2d-app demo but the output file is 0kb

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

Hardware (Jetson Nano 4G)

deepstream-app version 6.0.0

DeepStreamSDK 6.0.0

CUDA Driver Version: 10.2

CUDA Runtime Version: 10.2

TensorRT Version: 8.0

cuDNN Version: 8.2

I clone the project by this:
nvidia@nano:~/ds6_test$ git clone -b release/tao3.0_ds6.0ga GitHub - NVIDIA-AI-IOT/deepstream_tao_apps: Sample apps to demonstrate how to deploy models trained with TAO on DeepStream
and download the model.
Then I run referring the page:
image
nvidia@nano:~/ds6_test/deepstream_tao_apps/apps/tao_others/deepstream-bodypose2d-app$ ./deepstream-bodypose2d-app 1 …/…/…/configs/bodypose2d_tao/sample_bodypose2d_model_config.txt file:///opt/nvidia/deepstream/deepstream-6.0/samples/streams/sample_720p.jpg ./body2dout
When rebuilding the engine, it occurred some issues:

WARNING: [TRT]: Skipping tactic 21 due to oom error on requested size of 1728 detected for tactic -1218658103698133241.
ERROR: [TRT]: Tactic Device request: 1728MB Available: 1536MB. Device memory is insufficient to use tactic.
WARNING: [TRT]: Skipping tactic 22 due to oom error on requested size of 1728 detected for tactic -836875257600482091.
ERROR: [TRT]: Tactic Device request: 1728MB Available: 1536MB. Device memory is insufficient to use tactic.
WARNING: [TRT]: Skipping tactic 23 due to oom error on requested size of 1728 detected for tactic -410470605513481746.
ERROR: [TRT]: Tactic Device request: 1728MB Available: 1536MB. Device memory is insufficient to use tactic.
WARNING: [TRT]: Skipping tactic 0 due to oom error on requested size of 1728 detected for tactic -9153228964338181824.
ERROR: [TRT]: Tactic Device request: 1728MB Available: 1536MB. Device memory is insufficient to use tactic.
WARNING: [TRT]: Skipping tactic 1 due to oom error on requested size of 1728 detected for tactic -7394439838318485025.
ERROR: [TRT]: Tactic Device request: 1728MB Available: 1536MB. Device memory is insufficient to use tactic.
WARNING: [TRT]: Skipping tactic 0 due to oom error on requested size of 1728 detected for tactic 1002.
ERROR: [TRT]: Tactic Device request: 1728MB Available: 1536MB. Device memory is insufficient to use tactic.
WARNING: [TRT]: Skipping tactic 1 due to oom error on requested size of 1728 detected for tactic 0.
WARNING: [TRT]: Internal error: cannot reformat, disabling format. Try decreasing the workspace size with IBuilderConfig::setMaxWorkspaceSize().
WARNING: [TRT]: Memory requirements of format conversion cannot be satisfied during timing, format rejected.
ERROR: [TRT]: Tactic Device request: 1728MB Available: 1536MB. Device memory is insufficient to use tactic.
WARNING: [TRT]: Skipping tactic 0 due to oom error on requested size of 1728 detected for tactic 1002.
ERROR: [TRT]: Tactic Device request: 1728MB Available: 1536MB. Device memory is insufficient to use tactic.
WARNING: [TRT]: Skipping tactic 1 due to oom error on requested size of 1728 detected for tactic 0.
WARNING: [TRT]: Internal error: cannot reformat, disabling format. Try decreasing the workspace size with IBuilderConfig::setMaxWorkspaceSize().
WARNING: [TRT]: Memory requirements of format conversion cannot be satisfied during timing, format rejected.
0:23:43.027788485  8437   0x5574cac830 INFO                 nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger:<primary-infer-engine1> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1947> [UID = 1]: serialize cuda engine to file: /opt/nvidia/deepstream/deepstream-6.0/sources/apps/sample_apps/deepstream_tao_apps/models/bodypose2d/model.etlt_b32_gpu0_fp16.engine successfully
INFO: [FullDims Engine Info]: layers num: 3
0   INPUT  kFLOAT input_1:0       288x384x3       min: 1x288x384x3     opt: 32x288x384x3    Max: 32x288x384x3    
1   OUTPUT kFLOAT heatmap_out/BiasAdd:0 36x48x19        min: 0               opt: 0               Max: 0               
2   OUTPUT kFLOAT conv2d_transpose_1/BiasAdd:0 144x192x38      min: 0               opt: 0               Max: 0               

Killed

I think it should be low memory, but finally I think it runs successful.
However, the output file is just 0kb, it is just an empty file.

image
image730×266 120 KB

Is there any way to fix it? THx.

2

And I try to run another app,
nvidia@nano:~/ds6_test/deepstream_tao_apps/apps/tao_others/deepstream-gesture-app$ ./deepstream-gesture-app 2 …/…/…/configs/bodypose2d_tao/sample_bodypose2d_model_config.txt file:///opt/nvidia/deepstream/deepstream-6.0/samples/streams/sample_720p.jpg ./gesture
it just output the usage.

image
image727×156 76.8 KB

Is there a problem with my operation?

3

From the log, it is “Killed”. So, it does not run successfully.

Please refer to https://github.com/NVIDIA-AI-IOT/deepstream_tao_apps/tree/master/apps/tao_others
A sample of gesture:

./deepstream-gesture-app 1 1 ../../../configs/bodypose2d_tao/sample_bodypose2d_model_config.txt file:///usr/data/bodypose2d_test.png ./gesture

4

Thanks for your reply.
The engine is built successfully, but when I run again, it just displays killed.

nvidia@nano:~/ds6_test/deepstream_tao_apps/apps/tao_others/deepstream-bodypose2d-app$  ./deepstream-bodypose2d-app 1 ../../../configs/bodypose2d_tao/sample_bodypose2d_model_config.txt file:///opt/nvidia/deepstream/deepstream-6.0/samples/streams/sample_720p.jpg ./body2dout
Request sink_0 pad from streammux
joint Edges 1 , 8
joint Edges 8 , 9
joint Edges 9 , 10
joint Edges 1 , 11
joint Edges 11 , 12
joint Edges 12 , 13
joint Edges 1 , 2
joint Edges 2 , 3
joint Edges 3 , 4
joint Edges 2 , 16
joint Edges 1 , 5
joint Edges 5 , 6
joint Edges 6 , 7
joint Edges 5 , 17
joint Edges 1 , 0
joint Edges 0 , 14
joint Edges 0 , 15
joint Edges 14 , 16
joint Edges 15 , 17
connections 0 , 1
connections 1 , 2
connections 1 , 5
connections 2 , 3
connections 3 , 4
connections 5 , 6
connections 6 , 7
connections 2 , 8
connections 8 , 9
connections 9 , 10
connections 5 , 11
connections 11 , 12
connections 12 , 13
connections 0 , 14
connections 14 , 16
connections 8 , 11
connections 15 , 17
connections 0 , 15
Now playing: file:///opt/nvidia/deepstream/deepstream-6.0/samples/streams/sample_720p.jpg
0:00:06.187699234 20930   0x55c556c8d0 INFO                 nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger:<primary-infer-engine1> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1900> [UID = 1]: deserialized trt engine from :/opt/nvidia/deepstream/deepstream-6.0/sources/apps/sample_apps/deepstream_tao_apps/models/bodypose2d/model.etlt_b32_gpu0_fp16.engine
INFO: [FullDims Engine Info]: layers num: 3
0   INPUT  kFLOAT input_1:0       288x384x3       min: 1x288x384x3     opt: 32x288x384x3    Max: 32x288x384x3    
1   OUTPUT kFLOAT heatmap_out/BiasAdd:0 36x48x19        min: 0               opt: 0               Max: 0               
2   OUTPUT kFLOAT conv2d_transpose_1/BiasAdd:0 144x192x38      min: 0               opt: 0               Max: 0               

0:00:06.187985022 20930   0x55c556c8d0 INFO                 nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger:<primary-infer-engine1> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2004> [UID = 1]: Use deserialized engine model: /opt/nvidia/deepstream/deepstream-6.0/sources/apps/sample_apps/deepstream_tao_apps/models/bodypose2d/model.etlt_b32_gpu0_fp16.engine
Killed

For gesture:
I am trying
./deepstream-gesture-app 1 1 …/…/…/configs/bodypose2d_tao/sample_bodypose2d_model_config.txt file:///opt/nvidia/deepstream/deepstream-6.0/samples/streams/sample_720p.jpg ./gesture
but the same result:

WARNING: [TRT]: Detected invalid timing cache, setup a local cache instead
0:00:52.364826100 20941   0x55ba8b14c0 INFO                 nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger:<second-infer-engine> NvDsInferContext[UID 2]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1947> [UID = 2]: serialize cuda engine to file: /opt/nvidia/deepstream/deepstream-6.0/sources/apps/sample_apps/deepstream_tao_apps/models/gesture/gesture.etlt_b8_gpu0_fp16.engine successfully
INFO: [FullDims Engine Info]: layers num: 2
0   INPUT  kFLOAT input_1         3x160x160       min: 1x3x160x160     opt: 8x3x160x160     Max: 8x3x160x160     
1   OUTPUT kFLOAT activation_18   6               min: 0               opt: 0               Max: 0               

0:00:52.430272170 20941   0x55ba8b14c0 INFO                 nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<second-infer-engine> [UID 2]: Load new model:../../../configs/gesture_tao/gesture_sgie_config.txt sucessfully
0:00:53.855999543 20941   0x55ba8b14c0 INFO                 nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger:<primary-infer-engine1> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1900> [UID = 1]: deserialized trt engine from :/opt/nvidia/deepstream/deepstream-6.0/sources/apps/sample_apps/deepstream_tao_apps/models/bodypose2d/model.etlt_b32_gpu0_fp16.engine
INFO: [FullDims Engine Info]: layers num: 3
0   INPUT  kFLOAT input_1:0       288x384x3       min: 1x288x384x3     opt: 32x288x384x3    Max: 32x288x384x3    
1   OUTPUT kFLOAT heatmap_out/BiasAdd:0 36x48x19        min: 0               opt: 0               Max: 0               
2   OUTPUT kFLOAT conv2d_transpose_1/BiasAdd:0 144x192x38      min: 0               opt: 0               Max: 0               

0:00:53.856160011 20941   0x55ba8b14c0 INFO                 nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger:<primary-infer-engine1> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2004> [UID = 1]: Use deserialized engine model: /opt/nvidia/deepstream/deepstream-6.0/sources/apps/sample_apps/deepstream_tao_apps/models/bodypose2d/model.etlt_b32_gpu0_fp16.engine
Killed
5

So the problem is the engine? How can I solve this problem?

6

It is due to out of memory.
Please check if you can kill some other application to improve the memory.
Or, please try to use dgpu machines or other Jetson devices.

7

OK. I try to close the desktop display and it runs successfully.

nvidia@nano:~/ds6_test/deepstream_tao_apps/apps/tao_others/deepstream-bodypose2d-app$ sudo init 3
nvidia@nano:~/ds6_test/deepstream_tao_apps/apps/tao_others/deepstream-bodypose2d-app$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/nvidia/deepstream/deepstream/lib/cvcore_libs
nvidia@nano:~/ds6_test/deepstream_tao_apps/apps/tao_others/deepstream-bodypose2d-app$ unset DISPLAY
nvidia@nano:~/ds6_test/deepstream_tao_apps/apps/tao_others/deepstream-bodypose2d-app$ ./deepstream-bodypose2d-app 1 ../../../configs/bodypose2d_tao/sample_bodypose2d_model_config.txt file:///opt/nvidia/deepstream/deepstream-6.0/samples/streams/original-image.png ./body2dout

image

but log on screen still is warning.
Then I try to recover the display, run

nvidia@nano:~/ds6_test/deepstream_tao_apps/apps/tao_others/deepstream-bodypose2d-app$ sudo init 5

Input the passwd, but some error occurred：

image
image720×426 76.9 KB

How can I fix this?
Reboot is useful, is there another solution？
Thx.

8

For “recover the display in Nano”, please create a new topic in Jetson Nano forum. Thanks.

9

OK，sorry.

10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.