YoloV3: Number of unused weights left : 587

[siddhusiddhartha25]

Sir, I am working on object detection using custom yolov3 model with deepstream python apps. When i ran the python script file, the weights are not completely loaded.For coco dataset with pretrained yolov3 model, it is working. But for custom trained yolov3 modelo, it is showing the following error.

Jetson nano Information:
L4T 32.5.1 [ JetPack 4.5.1 ]
Ubuntu 18.04.5 LTS
Kernel Version: 4.9.201-tegra
CUDA 10.2.89
CUDA Architecture: 5.3
OpenCV version: 4.1.1
OpenCV Cuda: NO
CUDNN: 8.0.0.180
TensorRT: 7.1.3.0
Vision Works: 1.6.0.501
VPI: ii libnvvpi1 1.0.15 arm64 NVIDIA Vision Programming Interface library
Vulcan: 1.2.70

Output log:
aaeon@aaeon:/opt/nvidia/deepstream/deepstream-5.1/sources/python_deepstream_apps/yolov3_bus$ python3 tracker1.py file:///opt/nvidia/deepstream/deepstream-5.1/samples/streams/sample_720p.h264
Creating Pipeline

Creating streamux

Creating source_bin 0

Creating source bin
source-bin-00
Creating Pgie

Creating tracker
Creating tiler

Creating nvvidconv

Creating nvosd

Creating transform

Creating Sink

Unknown or legacy key specified ‘is-classifier’ for group [property]
Adding elements to Pipeline

Linking elements in the Pipeline

Now playing…
1 : file:///opt/nvidia/deepstream/deepstream-5.1/samples/streams/sample_720p.h264
Starting pipeline

gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream-5.1/lib/libnvds_mot_klt.so
gstnvtracker: Optional NvMOT_RemoveStreams not implemented
gstnvtracker: Batch processing is OFF
gstnvtracker: Past frame output is OFF
ERROR: [TRT]: INVALID_ARGUMENT: getPluginCreator could not find plugin YoloLayer_TRT version 1
ERROR: [TRT]: safeDeserializationUtils.cpp (323) - Serialization Error in load: 0 (Cannot deserialize plugin since corresponding IPluginCreator not found in Plugin Registry)
ERROR: [TRT]: INVALID_STATE: std::exception
ERROR: [TRT]: INVALID_CONFIG: Deserialize the cuda engine failed.
ERROR: Deserialize engine failed from file: /opt/nvidia/deepstream/deepstream-5.1/sources/python_deepstream_apps/yolov3_bus/yolov3-416.trt
0:00:14.534313122 8276 0x30b1c70 WARN nvinfer gstnvinfer.cpp:616:gst_nvinfer_logger: NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1691> [UID = 1]: deserialize engine from file :/opt/nvidia/deepstream/deepstream-5.1/sources/python_deepstream_apps/yolov3_bus/yolov3-416.trt failed
0:00:14.534399053 8276 0x30b1c70 WARN nvinfer gstnvinfer.cpp:616:gst_nvinfer_logger: NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1798> [UID = 1]: deserialize backend context from engine from file :/opt/nvidia/deepstream/deepstream-5.1/sources/python_deepstream_apps/yolov3_bus/yolov3-416.trt failed, try rebuild
0:00:14.534431498 8276 0x30b1c70 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger: NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1716> [UID = 1]: Trying to create engine from model files
WARNING: INT8 not supported by platform. Trying FP16 mode.
Loading pre-trained weights…
Loading weights of yolov3 complete!
Total Number of weights read : 61576929
Loading pre-trained weights…
Loading weights of yolov3 complete!
Total Number of weights read : 61576929
Building Yolo network…
layer inp_size out_size weightPtr
(0) conv-bn-leaky 3 x 416 x 416 32 x 416 x 416 992
(1) conv-bn-leaky 32 x 416 x 416 64 x 208 x 208 19680
(2) conv-bn-leaky 64 x 208 x 208 32 x 208 x 208 21856
(3) conv-bn-leaky 32 x 208 x 208 64 x 208 x 208 40544
(4) skip 64 x 208 x 208 64 x 208 x 208 -
(5) conv-bn-leaky 64 x 208 x 208 128 x 104 x 104 114784
(6) conv-bn-leaky 128 x 104 x 104 64 x 104 x 104 123232
(7) conv-bn-leaky 64 x 104 x 104 128 x 104 x 104 197472
(8) skip 128 x 104 x 104 128 x 104 x 104 -
(9) conv-bn-leaky 128 x 104 x 104 64 x 104 x 104 205920
(10) conv-bn-leaky 64 x 104 x 104 128 x 104 x 104 280160
(11) skip 128 x 104 x 104 128 x 104 x 104 -
(12) conv-bn-leaky 128 x 104 x 104 256 x 52 x 52 576096
(13) conv-bn-leaky 256 x 52 x 52 128 x 52 x 52 609376
(14) conv-bn-leaky 128 x 52 x 52 256 x 52 x 52 905312
(15) skip 256 x 52 x 52 256 x 52 x 52 -
(16) conv-bn-leaky 256 x 52 x 52 128 x 52 x 52 938592
(17) conv-bn-leaky 128 x 52 x 52 256 x 52 x 52 1234528
(18) skip 256 x 52 x 52 256 x 52 x 52 -
(19) conv-bn-leaky 256 x 52 x 52 128 x 52 x 52 1267808
(20) conv-bn-leaky 128 x 52 x 52 256 x 52 x 52 1563744
(21) skip 256 x 52 x 52 256 x 52 x 52 -
(22) conv-bn-leaky 256 x 52 x 52 128 x 52 x 52 1597024
(23) conv-bn-leaky 128 x 52 x 52 256 x 52 x 52 1892960
(24) skip 256 x 52 x 52 256 x 52 x 52 -
(25) conv-bn-leaky 256 x 52 x 52 128 x 52 x 52 1926240
(26) conv-bn-leaky 128 x 52 x 52 256 x 52 x 52 2222176
(27) skip 256 x 52 x 52 256 x 52 x 52 -
(28) conv-bn-leaky 256 x 52 x 52 128 x 52 x 52 2255456
(29) conv-bn-leaky 128 x 52 x 52 256 x 52 x 52 2551392
(30) skip 256 x 52 x 52 256 x 52 x 52 -
(31) conv-bn-leaky 256 x 52 x 52 128 x 52 x 52 2584672
(32) conv-bn-leaky 128 x 52 x 52 256 x 52 x 52 2880608
(33) skip 256 x 52 x 52 256 x 52 x 52 -
(34) conv-bn-leaky 256 x 52 x 52 128 x 52 x 52 2913888
(35) conv-bn-leaky 128 x 52 x 52 256 x 52 x 52 3209824
(36) skip 256 x 52 x 52 256 x 52 x 52 -
(37) conv-bn-leaky 256 x 52 x 52 512 x 26 x 26 4391520
(38) conv-bn-leaky 512 x 26 x 26 256 x 26 x 26 4523616
(39) conv-bn-leaky 256 x 26 x 26 512 x 26 x 26 5705312
(40) skip 512 x 26 x 26 512 x 26 x 26 -
(41) conv-bn-leaky 512 x 26 x 26 256 x 26 x 26 5837408
(42) conv-bn-leaky 256 x 26 x 26 512 x 26 x 26 7019104
(43) skip 512 x 26 x 26 512 x 26 x 26 -
(44) conv-bn-leaky 512 x 26 x 26 256 x 26 x 26 7151200
(45) conv-bn-leaky 256 x 26 x 26 512 x 26 x 26 8332896
(46) skip 512 x 26 x 26 512 x 26 x 26 -
(47) conv-bn-leaky 512 x 26 x 26 256 x 26 x 26 8464992
(48) conv-bn-leaky 256 x 26 x 26 512 x 26 x 26 9646688
(49) skip 512 x 26 x 26 512 x 26 x 26 -
(50) conv-bn-leaky 512 x 26 x 26 256 x 26 x 26 9778784
(51) conv-bn-leaky 256 x 26 x 26 512 x 26 x 26 10960480
(52) skip 512 x 26 x 26 512 x 26 x 26 -
(53) conv-bn-leaky 512 x 26 x 26 256 x 26 x 26 11092576
(54) conv-bn-leaky 256 x 26 x 26 512 x 26 x 26 12274272
(55) skip 512 x 26 x 26 512 x 26 x 26 -
(56) conv-bn-leaky 512 x 26 x 26 256 x 26 x 26 12406368
(57) conv-bn-leaky 256 x 26 x 26 512 x 26 x 26 13588064
(58) skip 512 x 26 x 26 512 x 26 x 26 -
(59) conv-bn-leaky 512 x 26 x 26 256 x 26 x 26 13720160
(60) conv-bn-leaky 256 x 26 x 26 512 x 26 x 26 14901856
(61) skip 512 x 26 x 26 512 x 26 x 26 -
(62) conv-bn-leaky 512 x 26 x 26 1024 x 13 x 13 19624544
(63) conv-bn-leaky 1024 x 13 x 13 512 x 13 x 13 20150880
(64) conv-bn-leaky 512 x 13 x 13 1024 x 13 x 13 24873568
(65) skip 1024 x 13 x 13 1024 x 13 x 13 -
(66) conv-bn-leaky 1024 x 13 x 13 512 x 13 x 13 25399904
(67) conv-bn-leaky 512 x 13 x 13 1024 x 13 x 13 30122592
(68) skip 1024 x 13 x 13 1024 x 13 x 13 -
(69) conv-bn-leaky 1024 x 13 x 13 512 x 13 x 13 30648928
(70) conv-bn-leaky 512 x 13 x 13 1024 x 13 x 13 35371616
(71) skip 1024 x 13 x 13 1024 x 13 x 13 -
(72) conv-bn-leaky 1024 x 13 x 13 512 x 13 x 13 35897952
(73) conv-bn-leaky 512 x 13 x 13 1024 x 13 x 13 40620640
(74) skip 1024 x 13 x 13 1024 x 13 x 13 -
(75) conv-bn-leaky 1024 x 13 x 13 512 x 13 x 13 41146976
(76) conv-bn-leaky 512 x 13 x 13 1024 x 13 x 13 45869664
(77) conv-bn-leaky 1024 x 13 x 13 512 x 13 x 13 46396000
(78) conv-bn-leaky 512 x 13 x 13 1024 x 13 x 13 51118688
(79) conv-bn-leaky 1024 x 13 x 13 512 x 13 x 13 51645024
(80) conv-bn-leaky 512 x 13 x 13 1024 x 13 x 13 56367712
(81) conv-linear 1024 x 13 x 13 18 x 13 x 13 56386162
(82) yolo 18 x 13 x 13 18 x 13 x 13 56386162
(83) route - 512 x 13 x 13 56386162
(84) conv-bn-leaky 512 x 13 x 13 256 x 13 x 13 56518258
INFO: [TRT]: mm1_85: broadcasting input0 to make tensors conform, dims(input0)=[1,26,13][NONE] dims(input1)=[256,13,13][NONE].
INFO: [TRT]: mm2_85: broadcasting input1 to make tensors conform, dims(input0)=[256,26,13][NONE] dims(input1)=[1,13,26][NONE].
(85) upsample 256 x 13 x 13 256 x 26 x 26 -
(86) route - 768 x 26 x 26 56518258
(87) conv-bn-leaky 768 x 26 x 26 256 x 26 x 26 56715890
(88) conv-bn-leaky 256 x 26 x 26 512 x 26 x 26 57897586
(89) conv-bn-leaky 512 x 26 x 26 256 x 26 x 26 58029682
(90) conv-bn-leaky 256 x 26 x 26 512 x 26 x 26 59211378
(91) conv-bn-leaky 512 x 26 x 26 256 x 26 x 26 59343474
(92) conv-bn-leaky 256 x 26 x 26 512 x 26 x 26 60525170
(93) conv-linear 512 x 26 x 26 18 x 26 x 26 60534404
(94) yolo 18 x 26 x 26 18 x 26 x 26 60534404
(95) route - 256 x 26 x 26 60534404
(96) conv-bn-leaky 256 x 26 x 26 128 x 26 x 26 60567684
INFO: [TRT]: mm1_97: broadcasting input0 to make tensors conform, dims(input0)=[1,52,26][NONE] dims(input1)=[128,26,26][NONE].
INFO: [TRT]: mm2_97: broadcasting input1 to make tensors conform, dims(input0)=[128,52,26][NONE] dims(input1)=[1,26,52][NONE].
(97) upsample 128 x 26 x 26 128 x 52 x 52 -
(98) route - 384 x 52 x 52 60567684
(99) conv-bn-leaky 384 x 52 x 52 128 x 52 x 52 60617348
(100) conv-bn-leaky 128 x 52 x 52 256 x 52 x 52 60913284
(101) conv-bn-leaky 256 x 52 x 52 128 x 52 x 52 60946564
(102) conv-bn-leaky 128 x 52 x 52 256 x 52 x 52 61242500
(103) conv-bn-leaky 256 x 52 x 52 128 x 52 x 52 61275780
(104) conv-bn-leaky 128 x 52 x 52 256 x 52 x 52 61571716
(105) conv-linear 256 x 52 x 52 18 x 52 x 52 61576342
(106) yolo 18 x 52 x 52 18 x 52 x 52 61576342
Number of unused weights left : 587
python3: yolo.cpp:335: NvDsInferStatus Yolo::buildYoloNetwork(std::vector&, nvinfer1::INetworkDefinition&): Assertion `0’ failed.
Aborted (core dumped)

Please help me to sort out this error, I am enclosing screenshot of the error.
Thanks in advance

[mchi]

could you share the config files?

Thanks!

[siddhusiddhartha25]

Sir, It is showing error when i tried to upload .weights file and .trt file. So, I am sharing the google drive link where i uploaded all the files. Sorry for the inconvenience Sir. Below is the google drive link.
https://drive.google.com/drive/folders/1dOtJlz9Om2yRD-3nuwpkFsoKMZSgouku?usp=sharing

Thanks in advance

[mchi]

Tried your files on DS 5.1 and got below errors, so seems you have some other customization to nvdsinfer_custom_impl_Yolo besides the yolov3 model.
So, looks this issue should be caused by your customization and we don’t have enough info.

root@cfe5d9d7a71c:/opt/nvidia/deepstream/deepstream-5.1/sources/objectDetector_Yolo# deepstream-app -c deepstream_app_config_yoloV3.txt
Unknown or legacy key specified 'is-classifier' for group [property]
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream-5.1/lib/libnvds_mot_klt.so
gstnvtracker: Optional NvMOT_RemoveStreams not implemented
gstnvtracker: Batch processing is OFF
gstnvtracker: Past frame output is OFF
ERROR: ../nvdsinfer/nvdsinfer_model_builder.cpp:1523 Deserialize engine failed because file path: /opt/nvidia/deepstream/deepstream-5.1/sources/objectDetector_Yolo/yolov3-416.trt open error
0:00:00.800485008   500 0x55fe1992e890 WARN                 nvinfer gstnvinfer.cpp:616:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1691> [UID = 1]: deserialize engine from file :/opt/nvidia/deepstream/deepstream-5.1/sources/objectDetector_Yolo/yolov3-416.trt failed
0:00:00.800526757   500 0x55fe1992e890 WARN                 nvinfer gstnvinfer.cpp:616:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1798> [UID = 1]: deserialize backend context from engine from file :/opt/nvidia/deepstream/deepstream-5.1/sources/objectDetector_Yolo/yolov3-416.trt failed, try rebuild
0:00:00.800539910   500 0x55fe1992e890 INFO                 nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1716> [UID = 1]: Trying to create engine from model files
Loading pre-trained weights...
deepstream-app: trt_utils.cpp:97: std::vector<float> loadWeights(std::__cxx11::string, const string&): Assertion `file.gcount() == 4' failed.
Aborted (core dumped)