We are trying to use our new custom-trained peoplenet model with deepstream-occupancy-analytics app. When we load the resnet34_detector.trt.int8 engine to the configuration file and we start the app we get the following error:
(deepstream-test5-analytics:20807): GLib-CRITICAL **: 17:03:47.922: g_strchomp: assertion ‘string != NULL’ failed
Warning: ‘input-dims’ parameter has been deprecated. Use ‘infer-dims’ instead.
Unknown or legacy key specified ‘gie-unique-output-blob-names’ for group [property]
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream-5.1/lib/libnvds_nvdcf.so
gstnvtracker: Batch processing is OFF
gstnvtracker: Past frame output is OFF
ERROR: [TRT]: coreReadArchive.cpp (38) - Serialization Error in verifyHeader: 0 (Version tag does not match)
ERROR: [TRT]: INVALID_STATE: std::exception
ERROR: [TRT]: INVALID_CONFIG: Deserialize the cuda engine failed.
ERROR: Deserialize engine failed from file: /opt/nvidia/deepstream/deepstream-5.1/sources/apps/sample_apps/deepstream-occupancy-analytics-AG/config/Age_and_Gender/experiment_dir_final/resnet34_detector.trt.int8
0:00:20.206519313 20807 0x558cd596f0 WARN nvinfer gstnvinfer.cpp:616:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1691> [UID = 1]: deserialize engine from file :/opt/nvidia/deepstream/deepstream-5.1/sources/apps/sample_apps/deepstream-occupancy-analytics-AG/config/Age_and_Gender/experiment_dir_final/resnet34_detector.trt.int8 failed
0:00:20.206614472 20807 0x558cd596f0 WARN nvinfer gstnvinfer.cpp:616:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1798> [UID = 1]: deserialize backend context from engine from file :/opt/nvidia/deepstream/deepstream-5.1/sources/apps/sample_apps/deepstream-occupancy-analytics-AG/config/Age_and_Gender/experiment_dir_final/resnet34_detector.trt.int8 failed, try rebuild
0:00:20.206652389 20807 0x558cd596f0 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1716> [UID = 1]: Trying to create engine from model files
WARNING: INT8 not supported by platform. Trying FP16 mode.
ERROR: No output layers specified. Need atleast one output layer
ERROR: Failed to create network using custom network creation function
ERROR: Failed to get cuda engine from custom library API
0:00:20.207064951 20807 0x558cd596f0 ERROR nvinfer gstnvinfer.cpp:613:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1736> [UID = 1]: build engine file failed
Segmentation fault (core dumped)
We get the following error (with commented and uncommented model-engine-file as recommended):
(deepstream-test5-analytics:27608): GLib-CRITICAL **: 17:15:20.248: g_strchug: assertion ‘string != NULL’ failed
(deepstream-test5-analytics:27608): GLib-CRITICAL **: 17:15:20.248: g_strchomp: assertion ‘string != NULL’ failed
Warning: ‘input-dims’ parameter has been deprecated. Use ‘infer-dims’ instead.
Unknown or legacy key specified ‘gie-unique-output-blob-names’ for group [property]
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream-5.1/lib/libnvds_nvdcf.so
gstnvtracker: Batch processing is OFF
gstnvtracker: Past frame output is OFF
ERROR: [TRT]: 6: The engine plan file is generated on an incompatible device, expecting compute 5.3 got compute 8.0, please rebuild.
ERROR: [TRT]: 4: [runtime.cpp::deserializeCudaEngine::76] Error Code 4: Internal Error (Engine deserialization failed.)
ERROR: Deserialize engine failed from file: /opt/nvidia/deepstream/deepstream-6.0/sources/apps/sample_apps/deepstream-occupancy-analytics-AG/config/Age_and_Gender/experiment_dir_final/resnet34_detector.trt
0:00:01.951993209 27608 0x55a7e8a8c0 WARN nvinfer gstnvinfer.cpp:635:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1889> [UID = 1]: deserialize engine from file :/opt/nvidia/deepstream/deepstream-6.0/sources/apps/sample_apps/deepstream-occupancy-analytics-AG/config/Age_and_Gender/experiment_dir_final/resnet34_detector.trt failed
0:00:01.952137431 27608 0x55a7e8a8c0 WARN nvinfer gstnvinfer.cpp:635:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1996> [UID = 1]: deserialize backend context from engine from file :/opt/nvidia/deepstream/deepstream-6.0/sources/apps/sample_apps/deepstream-occupancy-analytics-AG/config/Age_and_Gender/experiment_dir_final/resnet34_detector.trt failed, try rebuild
0:00:01.952172483 27608 0x55a7e8a8c0 INFO nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1914> [UID = 1]: Trying to create engine from model files
ERROR: No output layers specified. Need atleast one output layer
ERROR: Failed to create network using custom network creation function
ERROR: Failed to get cuda engine from custom library API
0:00:01.953026407 27608 0x55a7e8a8c0 ERROR nvinfer gstnvinfer.cpp:632:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1934> [UID = 1]: build engine file failed
terminate called after throwing an instance of ‘nvinfer1::InternalError’
what(): Assertion mRefCount > 0 failed.
Aborted (core dumped)
It’s not quite clear to me what the issue exactly is. Could you please help.
We continue to work on the app and the updates are the following:
-When we switched to Deepstream 6 we encountered a TRT device memory allocation problem when generating the device-based trt engine which was the reason behind our previous issue ( as we were using the one we generated in the Detectnet TAO notebook)
-We generated then the trt engine on jetson nano with jetpack 4.5.1 and Deepstream 5.1 instead, using this command:
Issue solved. Inside the configuration file config_infer_primary_peoplenet.txtx, in the property group, the parameter tlt-model-key should be equal to the key (-k) used while training the model.