yolov3 plugin error after running ./trt-yolo-app

Number of unused weights left : 18446744073709163896
trt-yolo-app: yolo.cpp:379: void Yolo::createYOLOEngine(int, std::__cxx11::string, std::__cxx11::string, std::__cxx11::string, nvinfer1::DataType, Int8EntropyCalibrator*): Assertion `0’ failed.
Aborted (core dumped)

YOLO v3 is trained on Custom dataset having 12 classes.

Solved

glad to hear that, thanks for replying

can you tell how?

please make sure below params set the path correct,

in file /path to lib/lib/network_config.cpp

const std::string kDS_LIB_PATH = “/home/tse/work/deepstream/DeepStream_Release/sources/gst-plugins/Deepst
ream_DsExample_Plugin-master/sources/lib/”;
const std::string kMODELS_PATH = kDS_LIB_PATH + “models/”;
const std::string kDETECTION_RESULTS_PATH = “…/…/…/data/detections/”;
const std::string kCALIBRATION_SET = “…/…/…/data/calibration_images.txt”;
const std::string kTEST_IMAGES = “…/…/…/data/test_images.txt”;

and /path to data/data/test_images.txt, content inside use the correct path for the test image, and make sure
you have the test images under the path.

I made sure to add the correct path in the test_images.txt file.

This is the error I’m getting:

File does not exist : sources/lib/models/yolov3-kFLOAT-batch1.engine
Unable to find cached TensorRT engine for network : yolov3 precision : kFLOAT and batch size :1
Creating a new TensorRT Engine
Loading pre-trained weights…
Loading complete!
Total Number of weights read : 62001757
Unsupported layer type --> ““button” aria-label=“Switch branches or tags” aria-expanded=“false” aria-haspopup=“true”>”
trt-yolo-app: yolo.cpp:372: void Yolo::createYOLOEngine(int, std::__cxx11::string, std::__cxx11::string, std::__cxx11::string, nvinfer1::DataType, Int8EntropyCalibrator*): Assertion `0’ failed.
Aborted (core dumped)

I’m so sorry. I just realized I’m in the incorrect forum.

I’m working with a tegra device.

Apologies.

sources/lib/models/yolov3-kFLOAT-batch1.engine
check the path i mentioned in comment 5 in neteork_config.cpp,use absolute path,or if you use relative path,
please make sure it is correct

@rakhecha42,

Did you use wget to download your cfg files ? Because it seems to have the html source as well. Please double check your cfg file.

It got solved. Although thanks for checking

Hi, I got the same problem. I am using cuda10 tensorrt 5.0, and custom dataset with 13 classes. and no network_config.cpp in my repo. Can you explain how did you solve this problem?

Number of unused weights left : 18446744073709551615
trt-yolo-app: yolo.cpp:411: void Yolo::createYOLOEngine(nvinfer1::DataType, Int8EntropyCalibrator*): Assertion `0’ failed.
Aborted (core dumped)

@derekwong6666 I seems like an underflow issue. This may be happening because the number of weights in your weight file doesnt exactly match with the number of weights required for your network architecture. Can you post the entire log from the app ?

Loading pre-trained weights…
Loading complete!
Total Number of weights read : 61640961
layer inp_size out_size weightPtr
(1) conv-bn-leaky 3 x 416 x 416 32 x 416 x 416 992
(2) conv-bn-leaky 32 x 416 x 416 64 x 208 x 208 19680
(3) conv-bn-leaky 64 x 208 x 208 32 x 208 x 208 21856
(4) conv-bn-leaky 32 x 208 x 208 64 x 208 x 208 40544
(5) skip 64 x 208 x 208 64 x 208 x 208 -
(6) conv-bn-leaky 64 x 208 x 208 128 x 104 x 104 114784
(7) conv-bn-leaky 128 x 104 x 104 64 x 104 x 104 123232
(8) conv-bn-leaky 64 x 104 x 104 128 x 104 x 104 197472
(9) skip 128 x 104 x 104 128 x 104 x 104 -
(10) conv-bn-leaky 128 x 104 x 104 64 x 104 x 104 205920
(11) conv-bn-leaky 64 x 104 x 104 128 x 104 x 104 280160
(12) skip 128 x 104 x 104 128 x 104 x 104 -
(13) conv-bn-leaky 128 x 104 x 104 256 x 52 x 52 576096
(14) conv-bn-leaky 256 x 52 x 52 128 x 52 x 52 609376
(15) conv-bn-leaky 128 x 52 x 52 256 x 52 x 52 905312
(16) skip 256 x 52 x 52 256 x 52 x 52 -
(17) conv-bn-leaky 256 x 52 x 52 128 x 52 x 52 938592
(18) conv-bn-leaky 128 x 52 x 52 256 x 52 x 52 1234528
(19) skip 256 x 52 x 52 256 x 52 x 52 -
(20) conv-bn-leaky 256 x 52 x 52 128 x 52 x 52 1267808
(21) conv-bn-leaky 128 x 52 x 52 256 x 52 x 52 1563744
(22) skip 256 x 52 x 52 256 x 52 x 52 -
(23) conv-bn-leaky 256 x 52 x 52 128 x 52 x 52 1597024
(24) conv-bn-leaky 128 x 52 x 52 256 x 52 x 52 1892960
(25) skip 256 x 52 x 52 256 x 52 x 52 -
(26) conv-bn-leaky 256 x 52 x 52 128 x 52 x 52 1926240
(27) conv-bn-leaky 128 x 52 x 52 256 x 52 x 52 2222176
(28) skip 256 x 52 x 52 256 x 52 x 52 -
(29) conv-bn-leaky 256 x 52 x 52 128 x 52 x 52 2255456
(30) conv-bn-leaky 128 x 52 x 52 256 x 52 x 52 2551392
(31) skip 256 x 52 x 52 256 x 52 x 52 -
(32) conv-bn-leaky 256 x 52 x 52 128 x 52 x 52 2584672
(33) conv-bn-leaky 128 x 52 x 52 256 x 52 x 52 2880608
(34) skip 256 x 52 x 52 256 x 52 x 52 -
(35) conv-bn-leaky 256 x 52 x 52 128 x 52 x 52 2913888
(36) conv-bn-leaky 128 x 52 x 52 256 x 52 x 52 3209824
(37) skip 256 x 52 x 52 256 x 52 x 52 -
(38) conv-bn-leaky 256 x 52 x 52 512 x 26 x 26 4391520
(39) conv-bn-leaky 512 x 26 x 26 256 x 26 x 26 4523616
(40) conv-bn-leaky 256 x 26 x 26 512 x 26 x 26 5705312
(41) skip 512 x 26 x 26 512 x 26 x 26 -
(42) conv-bn-leaky 512 x 26 x 26 256 x 26 x 26 5837408
(43) conv-bn-leaky 256 x 26 x 26 512 x 26 x 26 7019104
(44) skip 512 x 26 x 26 512 x 26 x 26 -
(45) conv-bn-leaky 512 x 26 x 26 256 x 26 x 26 7151200
(46) conv-bn-leaky 256 x 26 x 26 512 x 26 x 26 8332896
(47) skip 512 x 26 x 26 512 x 26 x 26 -
(48) conv-bn-leaky 512 x 26 x 26 256 x 26 x 26 8464992
(49) conv-bn-leaky 256 x 26 x 26 512 x 26 x 26 9646688
(50) skip 512 x 26 x 26 512 x 26 x 26 -
(51) conv-bn-leaky 512 x 26 x 26 256 x 26 x 26 9778784
(52) conv-bn-leaky 256 x 26 x 26 512 x 26 x 26 10960480
(53) skip 512 x 26 x 26 512 x 26 x 26 -
(54) conv-bn-leaky 512 x 26 x 26 256 x 26 x 26 11092576
(55) conv-bn-leaky 256 x 26 x 26 512 x 26 x 26 12274272
(56) skip 512 x 26 x 26 512 x 26 x 26 -
(57) conv-bn-leaky 512 x 26 x 26 256 x 26 x 26 12406368
(58) conv-bn-leaky 256 x 26 x 26 512 x 26 x 26 13588064
(59) skip 512 x 26 x 26 512 x 26 x 26 -
(60) conv-bn-leaky 512 x 26 x 26 256 x 26 x 26 13720160
(61) conv-bn-leaky 256 x 26 x 26 512 x 26 x 26 14901856
(62) skip 512 x 26 x 26 512 x 26 x 26 -
(63) conv-bn-leaky 512 x 26 x 26 1024 x 13 x 13 19624544
(64) conv-bn-leaky 1024 x 13 x 13 512 x 13 x 13 20150880
(65) conv-bn-leaky 512 x 13 x 13 1024 x 13 x 13 24873568
(66) skip 1024 x 13 x 13 1024 x 13 x 13 -
(67) conv-bn-leaky 1024 x 13 x 13 512 x 13 x 13 25399904
(68) conv-bn-leaky 512 x 13 x 13 1024 x 13 x 13 30122592
(69) skip 1024 x 13 x 13 1024 x 13 x 13 -
(70) conv-bn-leaky 1024 x 13 x 13 512 x 13 x 13 30648928
(71) conv-bn-leaky 512 x 13 x 13 1024 x 13 x 13 35371616
(72) skip 1024 x 13 x 13 1024 x 13 x 13 -
(73) conv-bn-leaky 1024 x 13 x 13 512 x 13 x 13 35897952
(74) conv-bn-leaky 512 x 13 x 13 1024 x 13 x 13 40620640
(75) skip 1024 x 13 x 13 1024 x 13 x 13 -
(76) conv-bn-leaky 1024 x 13 x 13 512 x 13 x 13 41146976
(77) conv-bn-leaky 512 x 13 x 13 1024 x 13 x 13 45869664
(78) conv-bn-leaky 1024 x 13 x 13 512 x 13 x 13 46396000
(79) conv-bn-leaky 512 x 13 x 13 1024 x 13 x 13 51118688
(80) conv-bn-leaky 1024 x 13 x 13 512 x 13 x 13 51645024
(81) conv-bn-leaky 512 x 13 x 13 1024 x 13 x 13 56367712
(82) conv-linear 1024 x 13 x 13 54 x 13 x 13 56423062
(83) yolo 54 x 13 x 13 54 x 13 x 13 56423062
(84) route - 512 x 13 x 13 56423062
(85) conv-bn-leaky 512 x 13 x 13 256 x 13 x 13 56555158
(86) upsample 256 x 13 x 13 256 x 26 x 26 -
(87) route - 768 x 26 x 26 56555158
(88) conv-bn-leaky 768 x 26 x 26 256 x 26 x 26 56752790
(89) conv-bn-leaky 256 x 26 x 26 512 x 26 x 26 57934486
(90) conv-bn-leaky 512 x 26 x 26 256 x 26 x 26 58066582
(91) conv-bn-leaky 256 x 26 x 26 512 x 26 x 26 59248278
(92) conv-bn-leaky 512 x 26 x 26 256 x 26 x 26 59380374
(93) conv-bn-leaky 256 x 26 x 26 512 x 26 x 26 60562070
(94) conv-linear 512 x 26 x 26 54 x 26 x 26 60589772
(95) yolo 54 x 26 x 26 54 x 26 x 26 60589772
(96) route - 256 x 26 x 26 60589772
(97) conv-bn-leaky 256 x 26 x 26 128 x 26 x 26 60623052
(98) upsample 128 x 26 x 26 128 x 52 x 52 -
(99) route - 384 x 52 x 52 60623052
(100) conv-bn-leaky 384 x 52 x 52 128 x 52 x 52 60672716
(101) conv-bn-leaky 128 x 52 x 52 256 x 52 x 52 60968652
(102) conv-bn-leaky 256 x 52 x 52 128 x 52 x 52 61001932
(103) conv-bn-leaky 128 x 52 x 52 256 x 52 x 52 61297868
(104) conv-bn-leaky 256 x 52 x 52 128 x 52 x 52 61331148
(105) conv-bn-leaky 128 x 52 x 52 256 x 52 x 52 61627084
(106) conv-linear 256 x 52 x 52 54 x 52 x 52 61640962
(107) yolo 54 x 52 x 52 54 x 52 x 52 61640962
Number of unused weights left : 18446744073709551615
trt-yolo-app: yolo.cpp:411: void Yolo::createYOLOEngine(nvinfer1::DataType, Int8EntropyCalibrator*): Assertion `0’ failed.
Aborted (core dumped)

I am sure, my custom cfg and weight can run using darknet environment.
If I commen if (weights.size() != weightPtr), it can build and serialize engine, but no detection result.

As you can see the total number of weights available for the network are

Total Number of weights read : 61640961

and the network requires 61640962 weights, which is the last value of the weight ptr. Most probably you are ommitting additional 4 bytes in the header. You can fix this by modifying the loadWeights(…) function in trt_utils.cpp. Based on the network_type you have chosen reduce the number of ignored bytes by 4 and you should be able to create the network after that.

I fixed it. Thx!

I am just wondering if it is possible to run trt-yolo-app on video stream rather than on images. Thanks.

trt-yolo-app just support images for input for now.

Hi, I got the same problem. I am using cuda10 tensorrt 5.1.5, and custom dataset with 4 classes. and no network_config.cpp in my repo. Can you explain how did you solve this problem?

Number of unused weights left : 18446744073709550878
trt-yolo-app: yolo.cpp:411: void Yolo::createYOLOEngine(nvinfer1::DataType, Int8EntropyCalibrator*): Assertion `0’ failed.

Loading complete!
Total Number of weights read : 4036266
layer inp_size out_size weightPtr
(1) conv-bn-leaky 3 x 608 x 608 19 x 608 x 608 589
(2) conv-bn-leaky 19 x 608 x 608 32 x 304 x 304 6189
(3) conv-bn-leaky 32 x 304 x 304 25 x 304 x 304 7089
(4) conv-bn-leaky 25 x 304 x 304 32 x 304 x 304 14417
(5) skip 32 x 304 x 304 32 x 304 x 304 -
(6) conv-bn-leaky 32 x 304 x 304 64 x 152 x 152 33105
(7) conv-bn-leaky 64 x 152 x 152 25 x 152 x 152 34805
(8) conv-bn-leaky 25 x 152 x 152 64 x 152 x 152 49461
(9) skip 64 x 152 x 152 64 x 152 x 152 -
(10) conv-bn-leaky 64 x 152 x 152 39 x 152 x 152 52113
(11) conv-bn-leaky 39 x 152 x 152 64 x 152 x 152 74833
(12) skip 64 x 152 x 152 64 x 152 x 152 -
(13) conv-bn-leaky 64 x 152 x 152 128 x 76 x 76 149073
(14) conv-bn-leaky 128 x 76 x 76 27 x 76 x 76 152637
(15) conv-bn-leaky 27 x 76 x 76 128 x 76 x 76 184253
(16) skip 128 x 76 x 76 128 x 76 x 76 -
(17) conv-bn-leaky 128 x 76 x 76 39 x 76 x 76 189401
(18) conv-bn-leaky 39 x 76 x 76 128 x 76 x 76 234841
(19) skip 128 x 76 x 76 128 x 76 x 76 -
(20) conv-bn-leaky 128 x 76 x 76 45 x 76 x 76 240781
(21) conv-bn-leaky 45 x 76 x 76 128 x 76 x 76 293133
(22) skip 128 x 76 x 76 128 x 76 x 76 -
(23) conv-bn-leaky 128 x 76 x 76 49 x 76 x 76 299601
(24) conv-bn-leaky 49 x 76 x 76 128 x 76 x 76 356561
(25) skip 128 x 76 x 76 128 x 76 x 76 -
(26) conv-bn-leaky 128 x 76 x 76 38 x 76 x 76 361577
(27) conv-bn-leaky 38 x 76 x 76 128 x 76 x 76 405865
(28) skip 128 x 76 x 76 128 x 76 x 76 -
(29) conv-bn-leaky 128 x 76 x 76 46 x 76 x 76 411937
(30) conv-bn-leaky 46 x 76 x 76 128 x 76 x 76 465441
(31) skip 128 x 76 x 76 128 x 76 x 76 -
(32) conv-bn-leaky 128 x 76 x 76 49 x 76 x 76 471909
(33) conv-bn-leaky 49 x 76 x 76 128 x 76 x 76 528869
(34) skip 128 x 76 x 76 128 x 76 x 76 -
(35) conv-bn-leaky 128 x 76 x 76 51 x 76 x 76 535601
(36) conv-bn-leaky 51 x 76 x 76 128 x 76 x 76 594865
(37) skip 128 x 76 x 76 128 x 76 x 76 -
(38) conv-bn-leaky 128 x 76 x 76 256 x 38 x 38 890801
(39) conv-bn-leaky 256 x 38 x 38 38 x 38 x 38 900681
(40) conv-bn-leaky 38 x 38 x 38 256 x 38 x 38 989257
(41) skip 256 x 38 x 38 256 x 38 x 38 -
(42) conv-bn-leaky 256 x 38 x 38 34 x 38 x 38 998097
(43) conv-bn-leaky 34 x 38 x 38 256 x 38 x 38 1077457
(44) skip 256 x 38 x 38 256 x 38 x 38 -
(45) conv-bn-leaky 256 x 38 x 38 42 x 38 x 38 1088377
(46) conv-bn-leaky 42 x 38 x 38 256 x 38 x 38 1186169
(47) skip 256 x 38 x 38 256 x 38 x 38 -
(48) conv-bn-leaky 256 x 38 x 38 56 x 38 x 38 1200729
(49) conv-bn-leaky 56 x 38 x 38 256 x 38 x 38 1330777
(50) skip 256 x 38 x 38 256 x 38 x 38 -
(51) conv-bn-leaky 256 x 38 x 38 47 x 38 x 38 1342997
(52) conv-bn-leaky 47 x 38 x 38 256 x 38 x 38 1452309
(53) skip 256 x 38 x 38 256 x 38 x 38 -
(54) conv-bn-leaky 256 x 38 x 38 38 x 38 x 38 1462189
(55) conv-bn-leaky 38 x 38 x 38 256 x 38 x 38 1550765
(56) skip 256 x 38 x 38 256 x 38 x 38 -
(57) conv-bn-leaky 256 x 38 x 38 45 x 38 x 38 1562465
(58) conv-bn-leaky 45 x 38 x 38 256 x 38 x 38 1667169
(59) skip 256 x 38 x 38 256 x 38 x 38 -
(60) conv-bn-leaky 256 x 38 x 38 30 x 38 x 38 1674969
(61) conv-bn-leaky 30 x 38 x 38 256 x 38 x 38 1745113
(62) skip 256 x 38 x 38 256 x 38 x 38 -
(63) conv-bn-leaky 256 x 38 x 38 512 x 19 x 19 2926809
(64) conv-bn-leaky 512 x 19 x 19 48 x 19 x 19 2951577
(65) conv-bn-leaky 48 x 19 x 19 512 x 19 x 19 3174809
(66) skip 512 x 19 x 19 512 x 19 x 19 -
(67) conv-bn-leaky 512 x 19 x 19 41 x 19 x 19 3195965
(68) conv-bn-leaky 41 x 19 x 19 512 x 19 x 19 3386941
(69) skip 512 x 19 x 19 512 x 19 x 19 -
(70) conv-bn-leaky 512 x 19 x 19 31 x 19 x 19 3402937
(71) conv-bn-leaky 31 x 19 x 19 512 x 19 x 19 3547833
(72) skip 512 x 19 x 19 512 x 19 x 19 -
(73) conv-bn-leaky 512 x 19 x 19 22 x 19 x 19 3559185
(74) conv-bn-leaky 22 x 19 x 19 512 x 19 x 19 3662609
(75) skip 512 x 19 x 19 512 x 19 x 19 -
(76) conv-bn-leaky 512 x 19 x 19 50 x 19 x 19 3688409
(77) conv-bn-leaky 50 x 19 x 19 52 x 19 x 19 3712017
(78) conv-bn-leaky 52 x 19 x 19 50 x 19 x 19 3714817
(79) conv-bn-leaky 50 x 19 x 19 47 x 19 x 19 3736155
(80) conv-bn-leaky 47 x 19 x 19 54 x 19 x 19 3738909
(81) conv-bn-leaky 54 x 19 x 19 130 x 19 x 19 3802609
(82) conv-bn-leaky 130 x 19 x 19 27 x 19 x 19 3806227
(83) yolo 27 x 19 x 19 27 x 19 x 19 3806227
(84) route - 54 x 19 x 19 3806227
(85) conv-bn-leaky 54 x 19 x 19 128 x 19 x 19 3813651
(86) upsample 128 x 19 x 19 128 x 38 x 38 -
(87) route - 384 x 38 x 38 3813651
(88) conv-bn-leaky 384 x 38 x 38 47 x 38 x 38 3831887
(89) conv-bn-leaky 47 x 38 x 38 32 x 38 x 38 3845551
(90) conv-bn-leaky 32 x 38 x 38 38 x 38 x 38 3846919
(91) conv-bn-leaky 38 x 38 x 38 68 x 38 x 38 3870447
(92) conv-bn-leaky 68 x 38 x 38 64 x 38 x 38 3875055
(93) conv-bn-leaky 64 x 38 x 38 107 x 38 x 38 3937115
(94) conv-bn-leaky 107 x 38 x 38 27 x 38 x 38 3940112
(95) yolo 27 x 38 x 38 27 x 38 x 38 3940112
(96) route - 64 x 38 x 38 3940112
(97) conv-bn-leaky 64 x 38 x 38 128 x 38 x 38 3948816
(98) upsample 128 x 38 x 38 128 x 76 x 76 -
(99) route - 256 x 76 x 76 3948816
(100) conv-bn-leaky 256 x 76 x 76 55 x 76 x 76 3963116
(101) conv-bn-leaky 55 x 76 x 76 38 x 76 x 76 3982078
(102) conv-bn-leaky 38 x 76 x 76 24 x 76 x 76 3983086
(103) conv-bn-leaky 24 x 76 x 76 41 x 76 x 76 3992106
(104) conv-bn-leaky 41 x 76 x 76 46 x 76 x 76 3994176
(105) conv-bn-leaky 46 x 76 x 76 96 x 76 x 76 4034304
(106) conv-bn-leaky 96 x 76 x 76 27 x 76 x 76 4037004
(107) yolo 27 x 76 x 76 27 x 76 x 76 4037004
Number of unused weights left : 18446744073709550878
trt-yolo-app: /home/ai/TensorRT_yolo3_module/deepstream_reference_apps/yolo/lib/yolo.cpp:407: void Yolo::createYOLOEngine(nvinfer1::DataType, Int8EntropyCalibrator*): Assertion `0’ failed.
Aborted (core dumped)

Total Number of weights read : 4036266

I have chosen reduce the number of ignored bytes by 4 and you should be able to don’t create the network after that.

Please create a new thread with your setup information/logs/questions.