• Hardware Platform: Tesla T4 GPU
• DeepStream Version: 5.1.0
• TensorRT Version 7.2.3.4
• NVIDIA GPU Driver Version 455.32.00
• Issue Type: questions
Hi,
We were trying to run yolov3 using deepstream SDK for an rtsp feed.
Yolov3 engine creation broke after loading the weights.
The output logs are as follows:
LOGS:
Unknown or legacy key specified ‘is-classifier’ for group [property]
Warn: ‘threshold’ parameter has been deprecated. Use ‘pre-cluster-threshold’ instead.
ERROR: …/nvdsinfer/nvdsinfer_model_builder.cpp:1523 Deserialize engine failed because file path: /opt/engines/model_b1_gpu0_fp16.engine open error
0:00:00.899249146 20310 0x7f6600696630 WARN nvinfer gstnvinfer.cpp:616:gst_nvinfer_logger: NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::deserializeEngineAndBackend()
<nvdsinfer_context_impl.cpp:1691> [UID = 1]: deserialize engine from file :/opt/engines/model_b1_gpu0_fp16.engine failed
0:00:00.899288855 20310 0x7f6600696630 WARN nvinfer gstnvinfer.cpp:616:gst_nvinfer_logger: NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1798> [UID = 1]: deserialize backend context from engine from file :/opt/engines/model_b1_gpu0_fp16.engine failed, try rebuild
0:00:00.899304599 20310 0x7f6600696630 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger: NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1716> [UID = 1]: Trying to create engine from model files
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_nvdcf.so
gstnvtracker: Batch processing is ON
gstnvtracker: Past frame output is ON
[NvDCF][Warning] minTrackingConfidenceDuringInactive is deprecated
[NvDCF] Initialized
Loading pre-trained weights…
Loading weights of yolov3 complete!
Total Number of weights read : 62001757
Loading pre-trained weights…
Building Yolo network…
layer inp_size out_size weightPtr
(0) conv-bn-leaky 3 x 608 x 608 32 x 608 x 608 992
(1) conv-bn-leaky 32 x 608 x 608 64 x 304 x 304 19680
(2) conv-bn-leaky 64 x 304 x 304 32 x 304 x 304 21856
(3) conv-bn-leaky 32 x 304 x 304 64 x 304 x 304 40544
(4) skip 64 x 304 x 304 64 x 304 x 304 -
(5) conv-bn-leaky 64 x 304 x 304 128 x 152 x 152 114784
(6) conv-bn-leaky 128 x 152 x 152 64 x 152 x 152 123232
(7) conv-bn-leaky 64 x 152 x 152 128 x 152 x 152 197472
(8) skip 128 x 152 x 152 128 x 152 x 152 -
(9) conv-bn-leaky 128 x 152 x 152 64 x 152 x 152 205920
(10) conv-bn-leaky 64 x 152 x 152 128 x 152 x 152 280160
(11) skip 128 x 152 x 152 128 x 152 x 152 -
(12) conv-bn-leaky 128 x 152 x 152 256 x 76 x 76 576096
(13) conv-bn-leaky 256 x 76 x 76 128 x 76 x 76 609376
(14) conv-bn-leaky 128 x 76 x 76 256 x 76 x 76 905312
(15) skip 256 x 76 x 76 256 x 76 x 76 -
(16) conv-bn-leaky 256 x 76 x 76 128 x 76 x 76 938592
(17) conv-bn-leaky 128 x 76 x 76 256 x 76 x 76 1234528
(18) skip 256 x 76 x 76 256 x 76 x 76 -
(19) conv-bn-leaky 256 x 76 x 76 128 x 76 x 76 1267808
(20) conv-bn-leaky 128 x 76 x 76 256 x 76 x 76 1563744
(21) skip 256 x 76 x 76 256 x 76 x 76 -
(22) conv-bn-leaky 256 x 76 x 76 128 x 76 x 76 1597024
(23) conv-bn-leaky 128 x 76 x 76 256 x 76 x 76 1892960
(24) skip 256 x 76 x 76 256 x 76 x 76 -
(25) conv-bn-leaky 256 x 76 x 76 128 x 76 x 76 1926240
(26) conv-bn-leaky 128 x 76 x 76 256 x 76 x 76 2222176
(27) skip 256 x 76 x 76 256 x 76 x 76 -
(28) conv-bn-leaky 256 x 76 x 76 128 x 76 x 76 2255456
(29) conv-bn-leaky 128 x 76 x 76 256 x 76 x 76 2551392
(30) skip 256 x 76 x 76 256 x 76 x 76 -
(31) conv-bn-leaky 256 x 76 x 76 128 x 76 x 76 2584672
(32) conv-bn-leaky 128 x 76 x 76 256 x 76 x 76 2880608
(33) skip 256 x 76 x 76 256 x 76 x 76 -
(34) conv-bn-leaky 256 x 76 x 76 128 x 76 x 76 2913888
(35) conv-bn-leaky 128 x 76 x 76 256 x 76 x 76 3209824
(36) skip 256 x 76 x 76 256 x 76 x 76 -
(37) conv-bn-leaky 256 x 76 x 76 512 x 38 x 38 4391520
(38) conv-bn-leaky 512 x 38 x 38 256 x 38 x 38 4523616
(39) conv-bn-leaky 256 x 38 x 38 512 x 38 x 38 5705312
(40) skip 512 x 38 x 38 512 x 38 x 38 -
(41) conv-bn-leaky 512 x 38 x 38 256 x 38 x 38 5837408
(42) conv-bn-leaky 256 x 38 x 38 512 x 38 x 38 7019104
(43) skip 512 x 38 x 38 512 x 38 x 38 -
(44) conv-bn-leaky 512 x 38 x 38 256 x 38 x 38 7151200
(45) conv-bn-leaky 256 x 38 x 38 512 x 38 x 38 8332896
(46) skip 512 x 38 x 38 512 x 38 x 38 -
(47) conv-bn-leaky 512 x 38 x 38 256 x 38 x 38 8464992
(48) conv-bn-leaky 256 x 38 x 38 512 x 38 x 38 9646688
(49) skip 512 x 38 x 38 512 x 38 x 38 -
(50) conv-bn-leaky 512 x 38 x 38 256 x 38 x 38 9778784
(51) conv-bn-leaky 256 x 38 x 38 512 x 38 x 38 10960480
(52) skip 512 x 38 x 38 512 x 38 x 38 -
(53) conv-bn-leaky 512 x 38 x 38 256 x 38 x 38 11092576
(54) conv-bn-leaky 256 x 38 x 38 512 x 38 x 38 12274272
(55) skip 512 x 38 x 38 512 x 38 x 38 -
(56) conv-bn-leaky 512 x 38 x 38 256 x 38 x 38 12406368
(57) conv-bn-leaky 256 x 38 x 38 512 x 38 x 38 13588064
(58) skip 512 x 38 x 38 512 x 38 x 38 -
(59) conv-bn-leaky 512 x 38 x 38 256 x 38 x 38 13720160
(60) conv-bn-leaky 256 x 38 x 38 512 x 38 x 38 14901856
(61) skip 512 x 38 x 38 512 x 38 x 38 -
(62) conv-bn-leaky 512 x 38 x 38 1024 x 19 x 19 19624544
(63) conv-bn-leaky 1024 x 19 x 19 512 x 19 x 19 20150880
(64) conv-bn-leaky 512 x 19 x 19 1024 x 19 x 19 24873568
(65) skip 1024 x 19 x 19 1024 x 19 x 19 -
(66) conv-bn-leaky 1024 x 19 x 19 512 x 19 x 19 25399904
(67) conv-bn-leaky 512 x 19 x 19 1024 x 19 x 19 30122592
(68) skip 1024 x 19 x 19 1024 x 19 x 19 -
(69) conv-bn-leaky 1024 x 19 x 19 512 x 19 x 19 30648928
(70) conv-bn-leaky 512 x 19 x 19 1024 x 19 x 19 35371616
(71) skip 1024 x 19 x 19 1024 x 19 x 19 -
(72) conv-bn-leaky 1024 x 19 x 19 512 x 19 x 19 35897952
(73) conv-bn-leaky 512 x 19 x 19 1024 x 19 x 19 40620640
(74) skip 1024 x 19 x 19 1024 x 19 x 19 -
(75) conv-bn-leaky 1024 x 19 x 19 512 x 19 x 19 41146976
(76) conv-bn-leaky 512 x 19 x 19 1024 x 19 x 19 45869664
(77) conv-bn-leaky 1024 x 19 x 19 512 x 19 x 19 46396000
(78) conv-bn-leaky 512 x 19 x 19 1024 x 19 x 19 51118688
(79) conv-bn-leaky 1024 x 19 x 19 512 x 19 x 19 51645024
(80) conv-bn-leaky 512 x 19 x 19 1024 x 19 x 19 56367712
(81) conv-linear 1024 x 19 x 19 255 x 19 x 19 56629087
(82) yolo 255 x 19 x 19 255 x 19 x 19 56629087
(83) route - 512 x 19 x 19 56629087
(84) conv-bn-leaky 512 x 19 x 19 256 x 19 x 19 56761183
INFO: …/nvdsinfer/nvdsinfer_func_utils.cpp:39 [TRT]: mm1_85: broadcasting input0 to make tensors conform, dims(input0)=[1,38,19][NONE] dims(input1)=[256,19,19][NONE].
INFO: …/nvdsinfer/nvdsinfer_func_utils.cpp:39 [TRT]: mm2_85: broadcasting input1 to make tensors conform, dims(input0)=[256,38,19][NONE] dims(input1)=[1,19,38][NONE].
(85) upsample 256 x 19 x 19 256 x 38 x 38 -
(86) route - 768 x 38 x 38 56761183
(87) conv-bn-leaky 768 x 38 x 38 256 x 38 x 38 56958815
(88) conv-bn-leaky 256 x 38 x 38 512 x 38 x 38 58140511
(89) conv-bn-leaky 512 x 38 x 38 256 x 38 x 38 58272607
(90) conv-bn-leaky 256 x 38 x 38 512 x 38 x 38 59454303
(91) conv-bn-leaky 512 x 38 x 38 256 x 38 x 38 59586399
(92) conv-bn-leaky 256 x 38 x 38 512 x 38 x 38 60768095
(93) conv-linear 512 x 38 x 38 255 x 38 x 38 60898910
(94) yolo 255 x 38 x 38 255 x 38 x 38 60898910
(95) route - 256 x 38 x 38 60898910
(96) conv-bn-leaky 256 x 38 x 38 128 x 38 x 38 60932190
INFO: …/nvdsinfer/nvdsinfer_func_utils.cpp:39 [TRT]: mm1_97: broadcasting input0 to make tensors conform, dims(input0)=[1,76,38][NONE] dims(input1)=[128,38,38][NONE].
INFO: …/nvdsinfer/nvdsinfer_func_utils.cpp:39 [TRT]: mm2_97: broadcasting input1 to make tensors conform, dims(input0)=[128,76,38][NONE] dims(input1)=[1,38,76][NONE].
(97) upsample 128 x 38 x 38 128 x 76 x 76 -
(98) route - 384 x 76 x 76 60932190
(99) conv-bn-leaky 384 x 76 x 76 128 x 76 x 76 60981854
(100) conv-bn-leaky 128 x 76 x 76 256 x 76 x 76 61277790
(101) conv-bn-leaky 256 x 76 x 76 128 x 76 x 76 61311070
(102) conv-bn-leaky 128 x 76 x 76 256 x 76 x 76 61607006
(103) conv-bn-leaky 256 x 76 x 76 128 x 76 x 76 61640286
(104) conv-bn-leaky 128 x 76 x 76 256 x 76 x 76 61936222
(105) conv-linear 256 x 76 x 76 255 x 76 x 76 62001757
(106) yolo 255 x 76 x 76 255 x 76 x 76 62001757
Output yolo blob names :
yolo_83
yolo_95
yolo_107
Total number of yolo layers: 257
Building yolo network complete!
Building the TensorRT Engine…
terminate called after throwing an instance of ‘pwgen::PwgenException’
what(): Driver error: