Unknown behavior for retinaface mobilenet face detection in deepstream 6.2

barzanhayati · April 29, 2023, 8:41am

Description

I’m trying to do face detection via deepstream 6.2 but I confront with unexpected errors while I have not these errors or warnings in deepstream 6.0 . Does this problem related to the face detection engine or libnvds_nvmultiobjecttracker.so. It’s possible to resolve it without doing extra changes in face detection model?

Environment

Deepstream Version: 6.2
TensorRT Version: 8.5.2
GPU Type: RTX 4090
Nvidia Driver Version: 525
CUDA Version: V11.8.89
CUDNN Version: 8.8.0
Operating System + Version: Ubuntu 20.04
Python Version (if applicable): Python 3.8.10
TensorFlow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag):

Relevant Files

TensorRTx - Implementation of popular deep learning networks with TensorRT network definition API

RetinaFace Pth Models

Generated RetinaFace: Mobilenet engine via mentioned instruction in retinaface subdirectory.
retinaface.wts (3.7 MB)

Attention

I do this work in this environments successfully for deepstream 6.0

TensorRT Version: 8.2.0
GPU Type: 2080Ti, 1080Ti, RTX 3060, 1650 Ti
Nvidia Driver Version: 470
CUDA Version: V11.4.152
CUDNN Version: 8.2.4
Operating System + Version: Ubuntu 20.04 , Ubuntu 18.04

So, my problem is related to the newest environment RTX 4090.

Steps To Reproduce

At first I generate face detection engine via mentioned github repository and I got these messages by running these commands:

./retina_mnet -s

[04/29/2023-11:18:51] [W] [TRT] CUDA lazy loading is not enabled. Enabling it can significantly reduce device memory usage. See `CUDA_MODULE_LOADING` in https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#env-vars
[04/29/2023-11:18:51] [W] [TRT] The implicit batch dimension mode has been deprecated. Please create the network with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag whenever possible.
Loading weights: ../retinaface.wts
Building engine, please wait for a while...
[04/29/2023-11:21:16] [W] [TRT] TensorRT encountered issues when converting weights between types and that could affect accuracy.
[04/29/2023-11:21:16] [W] [TRT] If this is not the desired behavior, please modify the weights or retrain with regularization to adjust the magnitude of the weights.
[04/29/2023-11:21:16] [W] [TRT] Check verbose logs for the list of affected weights.
[04/29/2023-11:21:16] [W] [TRT] - 27 weights are affected by this issue: Detected subnormal FP16 values.
[04/29/2023-11:21:16] [W] [TRT] - 1 weights are affected by this issue: Detected values less than smallest positive FP16 subnormal value and converted them to the FP16 minimum subnormalized value.
Build engine successfully!
[04/29/2023-11:21:16] [E] [TRT] 3: [builder.cpp::~Builder::307] Error Code 3: API Usage Error (Parameter check failed at: optimizer/api/builder.cpp::~Builder::307, condition: mObjectCounter.use_count() == 1. Destroying a builder object before destroying objects it created leads to undefined behavior.
)

Anyway it generates an engine file with these warnings and I could run face detection for sample images

./retina_mnet -d

and these warning has been generated:

[04/29/2023-11:47:52] [W] [TRT] CUDA lazy loading is not enabled. Enabling it can significantly reduce device memory usage. See `CUDA_MODULE_LOADING` in https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#env-vars
134592us
number of detections -> 1445
 -> 6.83762
after nms -> 103

And it shows that engine has been loaded successfully.

When I use CUDA_MODULE_LOADING=LAZY in command

CUDA_MODULE_LOADING=LAZY ./retina_mnet -s

some of those warning removed :


[04/29/2023-11:51:27] [W] [TRT] The implicit batch dimension mode has been deprecated. Please create the network with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag whenever possible.
Loading weights: ../../retinaface.wts
Building engine, please wait for a while...
Build engine successfully!
[04/29/2023-11:52:09] [E] [TRT] 3: [builder.cpp::~Builder::307] Error Code 3: API Usage Error (Parameter check failed at: optimizer/api/builder.cpp::~Builder::307, condition: mObjectCounter.use_count() == 1. Destroying a builder object before destroying objects it created leads to undefined behavior.
)

while non of these warnings was generated in the previous environments.

Till now I have an engine that could do face detection and find landmarks properly.
But When I use this engine in deepstream pipeline and I do compiling these warnings will be generated:

/home/nvs/Documents/face-recognition/resources/retinaface_post_processing/decode.h(74):-1: error: warning #997-D: function "nvinfer1::IPluginV2Ext::configurePlugin(const nvinfer1::Dims *, int32_t, const nvinfer1::Dims *, int32_t, const nvinfer1::DataType *, const nvinfer1::DataType *, const __nv_bool *, const __nv_bool *, nvinfer1::PluginFormat, int32_t)" is hidden by "nvinfer1::DecodePlugin::configurePlugin" -- virtual function override intended?

/home/nvs/Documents/face-recognition/resources/retinaface_post_processing/decode.h(74):-1: error: warning #997-D: function "nvinfer1::IPluginV2Ext::configurePlugin(const nvinfer1::Dims *, int32_t, const nvinfer1::Dims *, int32_t, const nvinfer1::DataType *, const nvinfer1::DataType *, const bool *, const bool *, nvinfer1::PluginFormat, int32_t)" is hidden by "nvinfer1::DecodePlugin::configurePlugin" -- virtual function override intended?

And I got these warnings or errors by running project:

gstnvtracker: Loading low-level lib at ../../resources/tracker_configs/libnvds_nvmultiobjecttracker.so
gstnvtracker: Batch processing is ON
gstnvtracker: Past frame output is OFF
~~ CLOG[src/modules/ReID/ReID.cpp, loadTRTEngine() @line 418]: Engine file does not exist
[NvMultiObjectTracker] Load engine failed. Create engine again.

!![ERROR] UFF file does not exist
[NvMultiObjectTracker] De-initialized
An exception occurred. UFF file does not exist
gstnvtracker: Failed to initialize tracker context!
gstnvtracker:: Failed to create batch context. Shutting down processing.
 Running... 
gstnvtracker: Loading low-level lib at ../../resources/tracker_configs/libnvds_nvmultiobjecttracker.so

gstnvtracker: Batch processing is ON
gstnvtracker: Past frame output is OFF
~~ CLOG[src/modules/ReID/ReID.cpp, loadTRTEngine() @line 418]: Engine file does not exist
[NvMultiObjectTracker] Load engine failed. Create engine again.

!![ERROR] UFF file does not exist
[NvMultiObjectTracker] De-initialized
An exception occurred. UFF file does not exist
gstnvtracker: Failed to initialize tracker context!
gstnvtracker:: Failed to create batch context. Shutting down processing.
gstnvtracker: Loading low-level lib at ../../resources/tracker_configs/libnvds_nvmultiobjecttracker.so
gstnvtracker: Batch processing is ON
gstnvtracker: Past frame output is OFF
~~ CLOG[src/modules/ReID/ReID.cpp, loadTRTEngine() @line 418]: Engine file does not exist
[NvMultiObjectTracker] Load engine failed. Create engine again.

!![ERROR] UFF file does not exist
[NvMultiObjectTracker] De-initialized
An exception occurred. UFF file does not exist
gstnvtracker: Failed to initialize tracker context!
gstnvtracker:: Failed to create batch context. Shutting down processing.
WARNING: [TRT]: CUDA lazy loading is not enabled. Enabling it can significantly reduce device memory usage. See `CUDA_MODULE_LOADING` in https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#env-vars
WARNING: [TRT]: CUDA lazy loading is not enabled. Enabling it can significantly reduce device memory usage. See `CUDA_MODULE_LOADING` in https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#env-vars
0:00:01.165972981 24867 0x563d811ce360 INFO                 nvinfer gstnvinfer.cpp:680:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1909> [UID = 1]: deserialized trt engine from :/home/nvs/Documents/face-recognition/resources/retina_mnet_FP32.engine
INFO: ../nvdsinfer/nvdsinfer_model_builder.cpp:610 [Implicit Engine Info]: layers num: 2
0   INPUT  kFLOAT data            3x480x640       
1   OUTPUT kFLOAT prob            189001x1x1      

0:00:01.205567514 24867 0x563d811ce360 INFO                 nvinfer gstnvinfer.cpp:680:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2012> [UID = 1]: Use deserialized engine model: /home/nvs/Documents/face-recognition/resources/retina_mnet_FP32.engine
0:00:01.274701668 24867 0x563d811ce360 INFO                 nvinfer gstnvinfer_impl.cpp:328:notifyLoadModelStatus:<primary-nvinference-engine> [UID 1]: Load new model:../../resources/config_infer_primary_FP32.txt sucessfully

And bysetting CUDA_MODULE_LOADING=LAZY

gstnvtracker: Loading low-level lib at ../../resources/tracker_configs/libnvds_nvmultiobjecttracker.so
gstnvtracker: Batch processing is ON
gstnvtracker: Past frame output is OFF
~~ CLOG[src/modules/ReID/ReID.cpp, loadTRTEngine() @line 418]: Engine file does not exist
[NvMultiObjectTracker] Load engine failed. Create engine again.

!![ERROR] UFF file does not exist
[NvMultiObjectTracker] De-initialized
An exception occurred. UFF file does not exist
gstnvtracker: Failed to initialize tracker context!
gstnvtracker:: Failed to create batch context. Shutting down processing.
gstnvtracker: Loading low-level lib at ../../resources/tracker_configs/libnvds_nvmultiobjecttracker.so

~~ CLOG[src/modules/ReID/ReID.cpp, loadTRTEngine() @line 418]: Engine file does not exist
[NvMultiObjectTracker] Load engine failed. Create engine again.

!![ERROR] UFF file does not exist
[NvMultiObjectTracker] De-initialized
An exception occurred. UFF file does not exist
gstnvtracker: Failed to initialize tracker context!
gstnvtracker:: Failed to create batch context. Shutting down processing.
gstnvtracker: Loading low-level lib at ../../resources/tracker_configs/libnvds_nvmultiobjecttracker.so
gstnvtracker: Batch processing is ON
gstnvtracker: Past frame output is OFF
~~ CLOG[src/modules/ReID/ReID.cpp, loadTRTEngine() @line 418]: Engine file does not exist
[NvMultiObjectTracker] Load engine failed. Create engine again.

!![ERROR] UFF file does not exist
[NvMultiObjectTracker] De-initialized
An exception occurred. UFF file does not exist
gstnvtracker: Failed to initialize tracker context!
gstnvtracker:: Failed to create batch context. Shutting down processing.
0:00:01.110999635 25010 0x557bad365360 INFO                 nvinfer gstnvinfer.cpp:680:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1909> [UID = 1]: deserialized trt engine from :/home/nvs/Documents/face-recognition/resources/retina_mnet_FP32.engine
INFO: ../nvdsinfer/nvdsinfer_model_builder.cpp:610 [Implicit Engine Info]: layers num: 2
0   INPUT  kFLOAT data            3x480x640       
1   OUTPUT kFLOAT prob            189001x1x1      

0:00:01.149862256 25010 0x557bad365360 INFO                 nvinfer gstnvinfer.cpp:680:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2012> [UID = 1]: Use deserialized engine model: /home/nvs/Documents/face-recognition/resources/retina_mnet_FP32.engine
0:00:01.218742237 25010 0x557bad365360 INFO                 nvinfer gstnvinfer_impl.cpp:328:notifyLoadModelStatus:<primary-nvinference-engine> [UID 1]: Load new model:../../resources/config_infer_primary_FP32.txt sucessfully
``

AakankshaS · April 29, 2023, 9:07am

Hi,

This looks like a Deepstream related issue. We will move this post to the Deepstream forum.

Thanks!

barzanhayati · July 9, 2023, 2:05pm

Have you check this challenges and does NVidia has any plan to resolve that?

Topic		Replies	Views
Deepstream 6.2 nvdcf tracker DeepStream SDK	29	1120	April 29, 2024
Issue with Deepstream Pose Estimation App: the "Decodebin did not pick nvidia decoder plugin" error in GStreamer DeepStream SDK gstreamer	12	1269	March 29, 2023
Is there anything that needs extra attention when using my own engine file? DeepStream SDK tensorrt , nvbugs	18	1352	July 20, 2021
Deepstream python sample"deepstream-test1" does not run DeepStream SDK cuda , gstreamer	10	386	September 5, 2023
Issues with running inference on multiple rtsp streams in deepstream-imagedata-multistream DeepStream SDK jetson-inference	24	670	August 7, 2024
Gaze Estimation App on deepstream 6.0: Engine File Error DeepStream SDK tensorrt , deepstream	5	32	October 25, 2024
Deepstream6.4 in Ubuntu24 Pipeline Not running DeepStream SDK	19	340	July 23, 2024
Runtime errors when running the human pose estimation application DeepStream SDK tensorrt , cuda , ubuntu	25	4827	October 12, 2021
DeepStream6.1 error in running sample app DeepStream SDK	5	656	November 21, 2022
Engine from file :primary.resnet10.caffemodel_b4_gpu0_int8.engine failed DeepStream SDK jetson-inference , docker , deepstream , graph-composer	8	509	March 7, 2023