Description
I’m trying to do face detection via deepstream 6.2 but I confront with unexpected errors while I have not these errors or warnings in deepstream 6.0 . Does this problem related to the face detection engine or libnvds_nvmultiobjecttracker.so. It’s possible to resolve it without doing extra changes in face detection model?
Environment
Deepstream Version: 6.2
TensorRT Version: 8.5.2
GPU Type: RTX 4090
Nvidia Driver Version: 525
CUDA Version: V11.8.89
CUDNN Version: 8.8.0
Operating System + Version: Ubuntu 20.04
Python Version (if applicable): Python 3.8.10
TensorFlow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag):
Relevant Files
TensorRTx - Implementation of popular deep learning networks with TensorRT network definition API
Generated RetinaFace: Mobilenet engine via mentioned instruction in retinaface subdirectory.
retinaface.wts (3.7 MB)
Attention
I do this work in this environments successfully for deepstream 6.0
TensorRT Version: 8.2.0
GPU Type: 2080Ti, 1080Ti, RTX 3060, 1650 Ti
Nvidia Driver Version: 470
CUDA Version: V11.4.152
CUDNN Version: 8.2.4
Operating System + Version: Ubuntu 20.04 , Ubuntu 18.04
So, my problem is related to the newest environment RTX 4090.
Steps To Reproduce
- At first I generate face detection engine via mentioned github repository and I got these messages by running these commands:
./retina_mnet -s
[04/29/2023-11:18:51] [W] [TRT] CUDA lazy loading is not enabled. Enabling it can significantly reduce device memory usage. See `CUDA_MODULE_LOADING` in https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#env-vars
[04/29/2023-11:18:51] [W] [TRT] The implicit batch dimension mode has been deprecated. Please create the network with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag whenever possible.
Loading weights: ../retinaface.wts
Building engine, please wait for a while...
[04/29/2023-11:21:16] [W] [TRT] TensorRT encountered issues when converting weights between types and that could affect accuracy.
[04/29/2023-11:21:16] [W] [TRT] If this is not the desired behavior, please modify the weights or retrain with regularization to adjust the magnitude of the weights.
[04/29/2023-11:21:16] [W] [TRT] Check verbose logs for the list of affected weights.
[04/29/2023-11:21:16] [W] [TRT] - 27 weights are affected by this issue: Detected subnormal FP16 values.
[04/29/2023-11:21:16] [W] [TRT] - 1 weights are affected by this issue: Detected values less than smallest positive FP16 subnormal value and converted them to the FP16 minimum subnormalized value.
Build engine successfully!
[04/29/2023-11:21:16] [E] [TRT] 3: [builder.cpp::~Builder::307] Error Code 3: API Usage Error (Parameter check failed at: optimizer/api/builder.cpp::~Builder::307, condition: mObjectCounter.use_count() == 1. Destroying a builder object before destroying objects it created leads to undefined behavior.
)
Anyway it generates an engine file with these warnings and I could run face detection for sample images
./retina_mnet -d
and these warning has been generated:
[04/29/2023-11:47:52] [W] [TRT] CUDA lazy loading is not enabled. Enabling it can significantly reduce device memory usage. See `CUDA_MODULE_LOADING` in https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#env-vars
134592us
number of detections -> 1445
-> 6.83762
after nms -> 103
And it shows that engine has been loaded successfully.
When I use CUDA_MODULE_LOADING=LAZY in command
CUDA_MODULE_LOADING=LAZY ./retina_mnet -s
some of those warning removed :
[04/29/2023-11:51:27] [W] [TRT] The implicit batch dimension mode has been deprecated. Please create the network with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag whenever possible.
Loading weights: ../../retinaface.wts
Building engine, please wait for a while...
Build engine successfully!
[04/29/2023-11:52:09] [E] [TRT] 3: [builder.cpp::~Builder::307] Error Code 3: API Usage Error (Parameter check failed at: optimizer/api/builder.cpp::~Builder::307, condition: mObjectCounter.use_count() == 1. Destroying a builder object before destroying objects it created leads to undefined behavior.
)
while non of these warnings was generated in the previous environments.
- Till now I have an engine that could do face detection and find landmarks properly.
- But When I use this engine in deepstream pipeline and I do compiling these warnings will be generated:
/home/nvs/Documents/face-recognition/resources/retinaface_post_processing/decode.h(74):-1: error: warning #997-D: function "nvinfer1::IPluginV2Ext::configurePlugin(const nvinfer1::Dims *, int32_t, const nvinfer1::Dims *, int32_t, const nvinfer1::DataType *, const nvinfer1::DataType *, const __nv_bool *, const __nv_bool *, nvinfer1::PluginFormat, int32_t)" is hidden by "nvinfer1::DecodePlugin::configurePlugin" -- virtual function override intended?
/home/nvs/Documents/face-recognition/resources/retinaface_post_processing/decode.h(74):-1: error: warning #997-D: function "nvinfer1::IPluginV2Ext::configurePlugin(const nvinfer1::Dims *, int32_t, const nvinfer1::Dims *, int32_t, const nvinfer1::DataType *, const nvinfer1::DataType *, const bool *, const bool *, nvinfer1::PluginFormat, int32_t)" is hidden by "nvinfer1::DecodePlugin::configurePlugin" -- virtual function override intended?
And I got these warnings or errors by running project:
gstnvtracker: Loading low-level lib at ../../resources/tracker_configs/libnvds_nvmultiobjecttracker.so
gstnvtracker: Batch processing is ON
gstnvtracker: Past frame output is OFF
~~ CLOG[src/modules/ReID/ReID.cpp, loadTRTEngine() @line 418]: Engine file does not exist
[NvMultiObjectTracker] Load engine failed. Create engine again.
!![ERROR] UFF file does not exist
[NvMultiObjectTracker] De-initialized
An exception occurred. UFF file does not exist
gstnvtracker: Failed to initialize tracker context!
gstnvtracker:: Failed to create batch context. Shutting down processing.
Running...
gstnvtracker: Loading low-level lib at ../../resources/tracker_configs/libnvds_nvmultiobjecttracker.so
gstnvtracker: Batch processing is ON
gstnvtracker: Past frame output is OFF
~~ CLOG[src/modules/ReID/ReID.cpp, loadTRTEngine() @line 418]: Engine file does not exist
[NvMultiObjectTracker] Load engine failed. Create engine again.
!![ERROR] UFF file does not exist
[NvMultiObjectTracker] De-initialized
An exception occurred. UFF file does not exist
gstnvtracker: Failed to initialize tracker context!
gstnvtracker:: Failed to create batch context. Shutting down processing.
gstnvtracker: Loading low-level lib at ../../resources/tracker_configs/libnvds_nvmultiobjecttracker.so
gstnvtracker: Batch processing is ON
gstnvtracker: Past frame output is OFF
~~ CLOG[src/modules/ReID/ReID.cpp, loadTRTEngine() @line 418]: Engine file does not exist
[NvMultiObjectTracker] Load engine failed. Create engine again.
!![ERROR] UFF file does not exist
[NvMultiObjectTracker] De-initialized
An exception occurred. UFF file does not exist
gstnvtracker: Failed to initialize tracker context!
gstnvtracker:: Failed to create batch context. Shutting down processing.
WARNING: [TRT]: CUDA lazy loading is not enabled. Enabling it can significantly reduce device memory usage. See `CUDA_MODULE_LOADING` in https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#env-vars
WARNING: [TRT]: CUDA lazy loading is not enabled. Enabling it can significantly reduce device memory usage. See `CUDA_MODULE_LOADING` in https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#env-vars
0:00:01.165972981 24867 0x563d811ce360 INFO nvinfer gstnvinfer.cpp:680:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1909> [UID = 1]: deserialized trt engine from :/home/nvs/Documents/face-recognition/resources/retina_mnet_FP32.engine
INFO: ../nvdsinfer/nvdsinfer_model_builder.cpp:610 [Implicit Engine Info]: layers num: 2
0 INPUT kFLOAT data 3x480x640
1 OUTPUT kFLOAT prob 189001x1x1
0:00:01.205567514 24867 0x563d811ce360 INFO nvinfer gstnvinfer.cpp:680:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2012> [UID = 1]: Use deserialized engine model: /home/nvs/Documents/face-recognition/resources/retina_mnet_FP32.engine
0:00:01.274701668 24867 0x563d811ce360 INFO nvinfer gstnvinfer_impl.cpp:328:notifyLoadModelStatus:<primary-nvinference-engine> [UID 1]: Load new model:../../resources/config_infer_primary_FP32.txt sucessfully
And bysetting CUDA_MODULE_LOADING=LAZY
gstnvtracker: Loading low-level lib at ../../resources/tracker_configs/libnvds_nvmultiobjecttracker.so
gstnvtracker: Batch processing is ON
gstnvtracker: Past frame output is OFF
~~ CLOG[src/modules/ReID/ReID.cpp, loadTRTEngine() @line 418]: Engine file does not exist
[NvMultiObjectTracker] Load engine failed. Create engine again.
!![ERROR] UFF file does not exist
[NvMultiObjectTracker] De-initialized
An exception occurred. UFF file does not exist
gstnvtracker: Failed to initialize tracker context!
gstnvtracker:: Failed to create batch context. Shutting down processing.
gstnvtracker: Loading low-level lib at ../../resources/tracker_configs/libnvds_nvmultiobjecttracker.so
~~ CLOG[src/modules/ReID/ReID.cpp, loadTRTEngine() @line 418]: Engine file does not exist
[NvMultiObjectTracker] Load engine failed. Create engine again.
!![ERROR] UFF file does not exist
[NvMultiObjectTracker] De-initialized
An exception occurred. UFF file does not exist
gstnvtracker: Failed to initialize tracker context!
gstnvtracker:: Failed to create batch context. Shutting down processing.
gstnvtracker: Loading low-level lib at ../../resources/tracker_configs/libnvds_nvmultiobjecttracker.so
gstnvtracker: Batch processing is ON
gstnvtracker: Past frame output is OFF
~~ CLOG[src/modules/ReID/ReID.cpp, loadTRTEngine() @line 418]: Engine file does not exist
[NvMultiObjectTracker] Load engine failed. Create engine again.
!![ERROR] UFF file does not exist
[NvMultiObjectTracker] De-initialized
An exception occurred. UFF file does not exist
gstnvtracker: Failed to initialize tracker context!
gstnvtracker:: Failed to create batch context. Shutting down processing.
0:00:01.110999635 25010 0x557bad365360 INFO nvinfer gstnvinfer.cpp:680:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1909> [UID = 1]: deserialized trt engine from :/home/nvs/Documents/face-recognition/resources/retina_mnet_FP32.engine
INFO: ../nvdsinfer/nvdsinfer_model_builder.cpp:610 [Implicit Engine Info]: layers num: 2
0 INPUT kFLOAT data 3x480x640
1 OUTPUT kFLOAT prob 189001x1x1
0:00:01.149862256 25010 0x557bad365360 INFO nvinfer gstnvinfer.cpp:680:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2012> [UID = 1]: Use deserialized engine model: /home/nvs/Documents/face-recognition/resources/retina_mnet_FP32.engine
0:00:01.218742237 25010 0x557bad365360 INFO nvinfer gstnvinfer_impl.cpp:328:notifyLoadModelStatus:<primary-nvinference-engine> [UID 1]: Load new model:../../resources/config_infer_primary_FP32.txt sucessfully
``