Unknown behavior for retinaface mobilenet face detection in deepstream 6.2

Description

I’m trying to do face detection via deepstream 6.2 but I confront with unexpected errors while I have not these errors or warnings in deepstream 6.0 . Does this problem related to the face detection engine or libnvds_nvmultiobjecttracker.so. It’s possible to resolve it without doing extra changes in face detection model?

Environment

Deepstream Version: 6.2
TensorRT Version: 8.5.2
GPU Type: RTX 4090
Nvidia Driver Version: 525
CUDA Version: V11.8.89
CUDNN Version: 8.8.0
Operating System + Version: Ubuntu 20.04
Python Version (if applicable): Python 3.8.10
TensorFlow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag):

Relevant Files

TensorRTx - Implementation of popular deep learning networks with TensorRT network definition API

RetinaFace Pth Models

Generated RetinaFace: Mobilenet engine via mentioned instruction in retinaface subdirectory.
retinaface.wts (3.7 MB)

Attention

I do this work in this environments successfully for deepstream 6.0

TensorRT Version: 8.2.0
GPU Type: 2080Ti, 1080Ti, RTX 3060, 1650 Ti
Nvidia Driver Version: 470
CUDA Version: V11.4.152
CUDNN Version: 8.2.4
Operating System + Version: Ubuntu 20.04 , Ubuntu 18.04

So, my problem is related to the newest environment RTX 4090.

Steps To Reproduce

./retina_mnet -s

[04/29/2023-11:18:51] [W] [TRT] CUDA lazy loading is not enabled. Enabling it can significantly reduce device memory usage. See `CUDA_MODULE_LOADING` in https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#env-vars
[04/29/2023-11:18:51] [W] [TRT] The implicit batch dimension mode has been deprecated. Please create the network with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag whenever possible.
Loading weights: ../retinaface.wts
Building engine, please wait for a while...
[04/29/2023-11:21:16] [W] [TRT] TensorRT encountered issues when converting weights between types and that could affect accuracy.
[04/29/2023-11:21:16] [W] [TRT] If this is not the desired behavior, please modify the weights or retrain with regularization to adjust the magnitude of the weights.
[04/29/2023-11:21:16] [W] [TRT] Check verbose logs for the list of affected weights.
[04/29/2023-11:21:16] [W] [TRT] - 27 weights are affected by this issue: Detected subnormal FP16 values.
[04/29/2023-11:21:16] [W] [TRT] - 1 weights are affected by this issue: Detected values less than smallest positive FP16 subnormal value and converted them to the FP16 minimum subnormalized value.
Build engine successfully!
[04/29/2023-11:21:16] [E] [TRT] 3: [builder.cpp::~Builder::307] Error Code 3: API Usage Error (Parameter check failed at: optimizer/api/builder.cpp::~Builder::307, condition: mObjectCounter.use_count() == 1. Destroying a builder object before destroying objects it created leads to undefined behavior.
)

Anyway it generates an engine file with these warnings and I could run face detection for sample images

./retina_mnet -d

and these warning has been generated:

[04/29/2023-11:47:52] [W] [TRT] CUDA lazy loading is not enabled. Enabling it can significantly reduce device memory usage. See `CUDA_MODULE_LOADING` in https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#env-vars
134592us
number of detections -> 1445
 -> 6.83762
after nms -> 103

And it shows that engine has been loaded successfully.

When I use CUDA_MODULE_LOADING=LAZY in command

CUDA_MODULE_LOADING=LAZY ./retina_mnet -s

some of those warning removed :


[04/29/2023-11:51:27] [W] [TRT] The implicit batch dimension mode has been deprecated. Please create the network with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag whenever possible.
Loading weights: ../../retinaface.wts
Building engine, please wait for a while...
Build engine successfully!
[04/29/2023-11:52:09] [E] [TRT] 3: [builder.cpp::~Builder::307] Error Code 3: API Usage Error (Parameter check failed at: optimizer/api/builder.cpp::~Builder::307, condition: mObjectCounter.use_count() == 1. Destroying a builder object before destroying objects it created leads to undefined behavior.
)

while non of these warnings was generated in the previous environments.

  • Till now I have an engine that could do face detection and find landmarks properly.
  • But When I use this engine in deepstream pipeline and I do compiling these warnings will be generated:
/home/nvs/Documents/face-recognition/resources/retinaface_post_processing/decode.h(74):-1: error: warning #997-D: function "nvinfer1::IPluginV2Ext::configurePlugin(const nvinfer1::Dims *, int32_t, const nvinfer1::Dims *, int32_t, const nvinfer1::DataType *, const nvinfer1::DataType *, const __nv_bool *, const __nv_bool *, nvinfer1::PluginFormat, int32_t)" is hidden by "nvinfer1::DecodePlugin::configurePlugin" -- virtual function override intended?

/home/nvs/Documents/face-recognition/resources/retinaface_post_processing/decode.h(74):-1: error: warning #997-D: function "nvinfer1::IPluginV2Ext::configurePlugin(const nvinfer1::Dims *, int32_t, const nvinfer1::Dims *, int32_t, const nvinfer1::DataType *, const nvinfer1::DataType *, const bool *, const bool *, nvinfer1::PluginFormat, int32_t)" is hidden by "nvinfer1::DecodePlugin::configurePlugin" -- virtual function override intended?

And I got these warnings or errors by running project:

gstnvtracker: Loading low-level lib at ../../resources/tracker_configs/libnvds_nvmultiobjecttracker.so
gstnvtracker: Batch processing is ON
gstnvtracker: Past frame output is OFF
~~ CLOG[src/modules/ReID/ReID.cpp, loadTRTEngine() @line 418]: Engine file does not exist
[NvMultiObjectTracker] Load engine failed. Create engine again.

!![ERROR] UFF file does not exist
[NvMultiObjectTracker] De-initialized
An exception occurred. UFF file does not exist
gstnvtracker: Failed to initialize tracker context!
gstnvtracker:: Failed to create batch context. Shutting down processing.
 Running... 
gstnvtracker: Loading low-level lib at ../../resources/tracker_configs/libnvds_nvmultiobjecttracker.so
gstnvtracker: Batch processing is ON
gstnvtracker: Past frame output is OFF
~~ CLOG[src/modules/ReID/ReID.cpp, loadTRTEngine() @line 418]: Engine file does not exist
[NvMultiObjectTracker] Load engine failed. Create engine again.

!![ERROR] UFF file does not exist
[NvMultiObjectTracker] De-initialized
An exception occurred. UFF file does not exist
gstnvtracker: Failed to initialize tracker context!
gstnvtracker:: Failed to create batch context. Shutting down processing.
gstnvtracker: Loading low-level lib at ../../resources/tracker_configs/libnvds_nvmultiobjecttracker.so
gstnvtracker: Batch processing is ON
gstnvtracker: Past frame output is OFF
~~ CLOG[src/modules/ReID/ReID.cpp, loadTRTEngine() @line 418]: Engine file does not exist
[NvMultiObjectTracker] Load engine failed. Create engine again.

!![ERROR] UFF file does not exist
[NvMultiObjectTracker] De-initialized
An exception occurred. UFF file does not exist
gstnvtracker: Failed to initialize tracker context!
gstnvtracker:: Failed to create batch context. Shutting down processing.
WARNING: [TRT]: CUDA lazy loading is not enabled. Enabling it can significantly reduce device memory usage. See `CUDA_MODULE_LOADING` in https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#env-vars
WARNING: [TRT]: CUDA lazy loading is not enabled. Enabling it can significantly reduce device memory usage. See `CUDA_MODULE_LOADING` in https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#env-vars
0:00:01.165972981 24867 0x563d811ce360 INFO                 nvinfer gstnvinfer.cpp:680:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1909> [UID = 1]: deserialized trt engine from :/home/nvs/Documents/face-recognition/resources/retina_mnet_FP32.engine
INFO: ../nvdsinfer/nvdsinfer_model_builder.cpp:610 [Implicit Engine Info]: layers num: 2
0   INPUT  kFLOAT data            3x480x640       
1   OUTPUT kFLOAT prob            189001x1x1      

0:00:01.205567514 24867 0x563d811ce360 INFO                 nvinfer gstnvinfer.cpp:680:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2012> [UID = 1]: Use deserialized engine model: /home/nvs/Documents/face-recognition/resources/retina_mnet_FP32.engine
0:00:01.274701668 24867 0x563d811ce360 INFO                 nvinfer gstnvinfer_impl.cpp:328:notifyLoadModelStatus:<primary-nvinference-engine> [UID 1]: Load new model:../../resources/config_infer_primary_FP32.txt sucessfully

And bysetting CUDA_MODULE_LOADING=LAZY

gstnvtracker: Loading low-level lib at ../../resources/tracker_configs/libnvds_nvmultiobjecttracker.so
gstnvtracker: Batch processing is ON
gstnvtracker: Past frame output is OFF
~~ CLOG[src/modules/ReID/ReID.cpp, loadTRTEngine() @line 418]: Engine file does not exist
[NvMultiObjectTracker] Load engine failed. Create engine again.

!![ERROR] UFF file does not exist
[NvMultiObjectTracker] De-initialized
An exception occurred. UFF file does not exist
gstnvtracker: Failed to initialize tracker context!
gstnvtracker:: Failed to create batch context. Shutting down processing.
gstnvtracker: Loading low-level lib at ../../resources/tracker_configs/libnvds_nvmultiobjecttracker.so
~~ CLOG[src/modules/ReID/ReID.cpp, loadTRTEngine() @line 418]: Engine file does not exist
[NvMultiObjectTracker] Load engine failed. Create engine again.

!![ERROR] UFF file does not exist
[NvMultiObjectTracker] De-initialized
An exception occurred. UFF file does not exist
gstnvtracker: Failed to initialize tracker context!
gstnvtracker:: Failed to create batch context. Shutting down processing.
gstnvtracker: Loading low-level lib at ../../resources/tracker_configs/libnvds_nvmultiobjecttracker.so
gstnvtracker: Batch processing is ON
gstnvtracker: Past frame output is OFF
~~ CLOG[src/modules/ReID/ReID.cpp, loadTRTEngine() @line 418]: Engine file does not exist
[NvMultiObjectTracker] Load engine failed. Create engine again.

!![ERROR] UFF file does not exist
[NvMultiObjectTracker] De-initialized
An exception occurred. UFF file does not exist
gstnvtracker: Failed to initialize tracker context!
gstnvtracker:: Failed to create batch context. Shutting down processing.
0:00:01.110999635 25010 0x557bad365360 INFO                 nvinfer gstnvinfer.cpp:680:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1909> [UID = 1]: deserialized trt engine from :/home/nvs/Documents/face-recognition/resources/retina_mnet_FP32.engine
INFO: ../nvdsinfer/nvdsinfer_model_builder.cpp:610 [Implicit Engine Info]: layers num: 2
0   INPUT  kFLOAT data            3x480x640       
1   OUTPUT kFLOAT prob            189001x1x1      

0:00:01.149862256 25010 0x557bad365360 INFO                 nvinfer gstnvinfer.cpp:680:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2012> [UID = 1]: Use deserialized engine model: /home/nvs/Documents/face-recognition/resources/retina_mnet_FP32.engine
0:00:01.218742237 25010 0x557bad365360 INFO                 nvinfer gstnvinfer_impl.cpp:328:notifyLoadModelStatus:<primary-nvinference-engine> [UID 1]: Load new model:../../resources/config_infer_primary_FP32.txt sucessfully
``

Hi,

This looks like a Deepstream related issue. We will move this post to the Deepstream forum.

Thanks!