Deepstream SDK for Yolov4 losing detections after 5 streams

Accelerated Computing Intelligent Video Analytics DeepStream SDK
,
1

I am losing detections on the nvdsinfer_custom_impl_Yolo by GitHub - marcoslucianops/DeepStream-Yolo: NVIDIA DeepStream SDK 6.3 / 6.2 / 6.1.1 / 6.1 / 6.0.1 / 6.0 / 5.1 implementation for YOLO models after 5 streams , but when i run the same yolov4 darknet models to convert to trt it works in this repo upto 16 GitHub - jkjung-avt/tensorrt_demos: TensorRT MODNet, YOLOv4, YOLOv3, SSD, MTCNN, and GoogLeNet for 12gb GPU
Deepstream version 6.2
Yolo V 4
Cuda 12.2
Runtime 11.8
Ubuntu 20.4
Gststreamer 1.16.0
This my yolo layers

Loading pre-trained weights

Loading weights of yolo-obj_best complete

Total weights read: 64003990

Building YOLO network

Layer Input Shape Output Shape WeightPtr

(0) conv_mish [3, 608, 608] [32, 608, 608] 992

(1) conv_mish [32, 608, 608] [64, 304, 304] 19680

(2) conv_mish [64, 304, 304] [64, 304, 304] 24032

(3) route: 1 - [64, 304, 304] -

(4) conv_mish [64, 304, 304] [64, 304, 304] 28384

(5) conv_mish [64, 304, 304] [32, 304, 304] 30560

(6) conv_mish [32, 304, 304] [64, 304, 304] 49248

(7) shortcut_linear: 4 [64, 304, 304] [64, 304, 304] -

(8) conv_mish [64, 304, 304] [64, 304, 304] 53600

(9) route: 8, 2 - [128, 304, 304] -

(10) conv_mish [128, 304, 304] [64, 304, 304] 62048

(11) conv_mish [64, 304, 304] [128, 152, 152] 136288

(12) conv_mish [128, 152, 152] [64, 152, 152] 144736

(13) route: 11 - [128, 152, 152] -

(14) conv_mish [128, 152, 152] [64, 152, 152] 153184

(15) conv_mish [64, 152, 152] [64, 152, 152] 157536

(16) conv_mish [64, 152, 152] [64, 152, 152] 194656

(17) shortcut_linear: 14 [64, 152, 152] [64, 152, 152] -

(18) conv_mish [64, 152, 152] [64, 152, 152] 199008

(19) conv_mish [64, 152, 152] [64, 152, 152] 236128

(20) shortcut_linear: 17 [64, 152, 152] [64, 152, 152] -

(21) conv_mish [64, 152, 152] [64, 152, 152] 240480

(22) route: 21, 12 - [128, 152, 152] -

(23) conv_mish [128, 152, 152] [128, 152, 152] 257376

(24) conv_mish [128, 152, 152] [256, 76, 76] 553312

(25) conv_mish [256, 76, 76] [128, 76, 76] 586592

(26) route: 24 - [256, 76, 76] -

(27) conv_mish [256, 76, 76] [128, 76, 76] 619872

(28) conv_mish [128, 76, 76] [128, 76, 76] 636768

(29) conv_mish [128, 76, 76] [128, 76, 76] 784736

(30) shortcut_linear: 27 [128, 76, 76] [128, 76, 76] -

(31) conv_mish [128, 76, 76] [128, 76, 76] 801632

(32) conv_mish [128, 76, 76] [128, 76, 76] 949600

(33) shortcut_linear: 30 [128, 76, 76] [128, 76, 76] -

(34) conv_mish [128, 76, 76] [128, 76, 76] 966496

(35) conv_mish [128, 76, 76] [128, 76, 76] 1114464

(36) shortcut_linear: 33 [128, 76, 76] [128, 76, 76] -

(37) conv_mish [128, 76, 76] [128, 76, 76] 1131360

(38) conv_mish [128, 76, 76] [128, 76, 76] 1279328

(39) shortcut_linear: 36 [128, 76, 76] [128, 76, 76] -

(40) conv_mish [128, 76, 76] [128, 76, 76] 1296224

(41) conv_mish [128, 76, 76] [128, 76, 76] 1444192

(42) shortcut_linear: 39 [128, 76, 76] [128, 76, 76] -

(43) conv_mish [128, 76, 76] [128, 76, 76] 1461088

(44) conv_mish [128, 76, 76] [128, 76, 76] 1609056

(45) shortcut_linear: 42 [128, 76, 76] [128, 76, 76] -

(46) conv_mish [128, 76, 76] [128, 76, 76] 1625952

(47) conv_mish [128, 76, 76] [128, 76, 76] 1773920

(48) shortcut_linear: 45 [128, 76, 76] [128, 76, 76] -

(49) conv_mish [128, 76, 76] [128, 76, 76] 1790816

(50) conv_mish [128, 76, 76] [128, 76, 76] 1938784

(51) shortcut_linear: 48 [128, 76, 76] [128, 76, 76] -

(52) conv_mish [128, 76, 76] [128, 76, 76] 1955680

(53) route: 52, 25 - [256, 76, 76] -

(54) conv_mish [256, 76, 76] [256, 76, 76] 2022240

(55) conv_mish [256, 76, 76] [512, 38, 38] 3203936

(56) conv_mish [512, 38, 38] [256, 38, 38] 3336032

(57) route: 55 - [512, 38, 38] -

(58) conv_mish [512, 38, 38] [256, 38, 38] 3468128

(59) conv_mish [256, 38, 38] [256, 38, 38] 3534688

(60) conv_mish [256, 38, 38] [256, 38, 38] 4125536

(61) shortcut_linear: 58 [256, 38, 38] [256, 38, 38] -

(62) conv_mish [256, 38, 38] [256, 38, 38] 4192096

(63) conv_mish [256, 38, 38] [256, 38, 38] 4782944

(64) shortcut_linear: 61 [256, 38, 38] [256, 38, 38] -

(65) conv_mish [256, 38, 38] [256, 38, 38] 4849504

(66) conv_mish [256, 38, 38] [256, 38, 38] 5440352

(67) shortcut_linear: 64 [256, 38, 38] [256, 38, 38] -

(68) conv_mish [256, 38, 38] [256, 38, 38] 5506912

(69) conv_mish [256, 38, 38] [256, 38, 38] 6097760

(70) shortcut_linear: 67 [256, 38, 38] [256, 38, 38] -

(71) conv_mish [256, 38, 38] [256, 38, 38] 6164320

(72) conv_mish [256, 38, 38] [256, 38, 38] 6755168

(73) shortcut_linear: 70 [256, 38, 38] [256, 38, 38] -

(74) conv_mish [256, 38, 38] [256, 38, 38] 6821728

(75) conv_mish [256, 38, 38] [256, 38, 38] 7412576

(76) shortcut_linear: 73 [256, 38, 38] [256, 38, 38] -

(77) conv_mish [256, 38, 38] [256, 38, 38] 7479136

(78) conv_mish [256, 38, 38] [256, 38, 38] 8069984

(79) shortcut_linear: 76 [256, 38, 38] [256, 38, 38] -

(80) conv_mish [256, 38, 38] [256, 38, 38] 8136544

(81) conv_mish [256, 38, 38] [256, 38, 38] 8727392

(82) shortcut_linear: 79 [256, 38, 38] [256, 38, 38] -

(83) conv_mish [256, 38, 38] [256, 38, 38] 8793952

(84) route: 83, 56 - [512, 38, 38] -

(85) conv_mish [512, 38, 38] [512, 38, 38] 9058144

(86) conv_mish [512, 38, 38] [1024, 19, 19] 13780832

(87) conv_mish [1024, 19, 19] [512, 19, 19] 14307168

(88) route: 86 - [1024, 19, 19] -

(89) conv_mish [1024, 19, 19] [512, 19, 19] 14833504

(90) conv_mish [512, 19, 19] [512, 19, 19] 15097696

(91) conv_mish [512, 19, 19] [512, 19, 19] 17459040

(92) shortcut_linear: 89 [512, 19, 19] [512, 19, 19] -

(93) conv_mish [512, 19, 19] [512, 19, 19] 17723232

(94) conv_mish [512, 19, 19] [512, 19, 19] 20084576

(95) shortcut_linear: 92 [512, 19, 19] [512, 19, 19] -

(96) conv_mish [512, 19, 19] [512, 19, 19] 20348768

(97) conv_mish [512, 19, 19] [512, 19, 19] 22710112

(98) shortcut_linear: 95 [512, 19, 19] [512, 19, 19] -

(99) conv_mish [512, 19, 19] [512, 19, 19] 22974304

(100) conv_mish [512, 19, 19] [512, 19, 19] 25335648

(101) shortcut_linear: 98 [512, 19, 19] [512, 19, 19] -

(102) conv_mish [512, 19, 19] [512, 19, 19] 25599840

(103) route: 102, 87 - [1024, 19, 19] -

(104) conv_mish [1024, 19, 19] [1024, 19, 19] 26652512

(105) conv_leaky [1024, 19, 19] [512, 19, 19] 27178848

(106) conv_leaky [512, 19, 19] [1024, 19, 19] 31901536

(107) conv_leaky [1024, 19, 19] [512, 19, 19] 32427872

(108) maxpool [512, 19, 19] [512, 19, 19] -

(109) route: 107 - [512, 19, 19] -

(110) maxpool [512, 19, 19] [512, 19, 19] -

(111) route: 107 - [512, 19, 19] -

(112) maxpool [512, 19, 19] [512, 19, 19] -

(113) route: 112, 110, 108, 107 - [2048, 19, 19] -

(114) conv_leaky [2048, 19, 19] [512, 19, 19] 33478496

(115) conv_leaky [512, 19, 19] [1024, 19, 19] 38201184

(116) conv_leaky [1024, 19, 19] [512, 19, 19] 38727520

(117) conv_leaky [512, 19, 19] [256, 19, 19] 38859616

(118) upsample [256, 19, 19] [256, 38, 38] -

(119) route: 85 - [512, 38, 38] -

(120) conv_leaky [512, 38, 38] [256, 38, 38] 38991712

(121) route: 120, 118 - [512, 38, 38] -

(122) conv_leaky [512, 38, 38] [256, 38, 38] 39123808

(123) conv_leaky [256, 38, 38] [512, 38, 38] 40305504

(124) conv_leaky [512, 38, 38] [256, 38, 38] 40437600

(125) conv_leaky [256, 38, 38] [512, 38, 38] 41619296

(126) conv_leaky [512, 38, 38] [256, 38, 38] 41751392

(127) conv_leaky [256, 38, 38] [128, 38, 38] 41784672

(128) upsample [128, 38, 38] [128, 76, 76] -

(129) route: 54 - [256, 76, 76] -

(130) conv_leaky [256, 76, 76] [128, 76, 76] 41817952

(131) route: 130, 128 - [256, 76, 76] -

(132) conv_leaky [256, 76, 76] [128, 76, 76] 41851232

(133) conv_leaky [128, 76, 76] [256, 76, 76] 42147168

(134) conv_leaky [256, 76, 76] [128, 76, 76] 42180448

(135) conv_leaky [128, 76, 76] [256, 76, 76] 42476384

(136) conv_leaky [256, 76, 76] [128, 76, 76] 42509664

(137) conv_leaky [128, 76, 76] [256, 76, 76] 42805600

(138) conv_linear [256, 76, 76] [18, 76, 76] 42810226

(139) yolo [18, 76, 76] - -

(140) route: 136 - [128, 76, 76] -

(141) conv_leaky [128, 76, 76] [256, 38, 38] 43106162

(142) route: 141, 126 - [512, 38, 38] -

(143) conv_leaky [512, 38, 38] [256, 38, 38] 43238258

(144) conv_leaky [256, 38, 38] [512, 38, 38] 44419954

(145) conv_leaky [512, 38, 38] [256, 38, 38] 44552050

(146) conv_leaky [256, 38, 38] [512, 38, 38] 45733746

(147) conv_leaky [512, 38, 38] [256, 38, 38] 45865842

(148) conv_leaky [256, 38, 38] [512, 38, 38] 47047538

(149) conv_linear [512, 38, 38] [18, 38, 38] 47056772

(150) yolo [18, 38, 38] - -

(151) route: 147 - [256, 38, 38] -

(152) conv_leaky [256, 38, 38] [512, 19, 19] 48238468

(153) route: 152, 116 - [1024, 19, 19] -

(154) conv_leaky [1024, 19, 19] [512, 19, 19] 48764804

(155) conv_leaky [512, 19, 19] [1024, 19, 19] 53487492

(156) conv_leaky [1024, 19, 19] [512, 19, 19] 54013828

(157) conv_leaky [512, 19, 19] [1024, 19, 19] 58736516

(158) conv_leaky [1024, 19, 19] [512, 19, 19] 59262852

(159) conv_leaky [512, 19, 19] [1024, 19, 19] 63985540

(160) conv_linear [1024, 19, 19] [18, 19, 19] 64003990

(161) yolo [18, 19, 19] - -

Output YOLO blob names:

yolo_140

yolo_151

yolo_162
Total number of YOLO layers: 507

pipeline
pipeline11767×1812 1.16 MB

This is config_infer_primary.txt
[property]
gpu-id=0
net-scale-factor=0.0039215697906911373
model-color-format=0
custom-network-config=models/yolo-obj.cfg
model-file=models/yolo-obj_best.weights
model-engine-file=model_b5_gpu0_fp32.engine
labelfile-path=labels.txt
batch-size=1
network-mode=0
num-detected-classes=1
interval=0
gie-unique-id=1
process-mode=1
network-type=0
cluster-mode=2
maintain-aspect-ratio=0
symmetric-padding=1
force-implicit-batch-dim=1
parse-bbox-func-name=NvDsInferParseYoloCuda
custom-lib-path=nvdsinfer_custom_impl_Yolo/libnvdsinfer_custom_impl_Yolo.so
engine-create-func-name=NvDsInferYoloCudaEngineGet
[class-attrs-all]
nms-iou-threshold=0.45
pre-cluster-threshold=0.25
nms-iou-threshold=0.3
topk=300

This is my pipeline
Please I am in a hurry could anyone help me asap, i would be greatful for them

2

Is this test also done in the pipeline as DeepStream-Yolo(only update model-engine-file in the config file)?

What’s the type of your GPU and how about the encoder&decoder usage when the error happens (nvidia-smi dmon)? There is nvv4l2h264enc in your DeepStream pipeline, not sure if this is the reason. Please refer to the “Max # of concurrent sessions” for encoder in table Video Encode and Decode GPU Support Matrix | NVIDIA Developer.
Please also share the full log for this issue, preferably addingGST_DEBUG=3 before the deepstream-app command line.

4

Q. Is this test also done in the pipeline as DeepStream-Yolo(only update model-engine-file in the config file)?
A. No, I have deepstream python apps for the sdk code, but only bboxparser from Deepstream-Yolo

Q. What’s the type of your GPU and how about the encoder&decoder usage when the error happens (nvidia-smi dmon)? There is nvv4l2h264enc in your DeepStream pipeline, not sure if this is the reason.
A. Yes I am nvv4l2h264enc in my Deepstream Pipeline, But Comaretively it larger still, But still why should detections be lost, encoder is not related to Detections righ.

I. Please refer to the “Max # of concurrent sessions” for encoder in table Video Encode and Decode GPU Support Matrix | NVIDIA Developer.
Well I am using A10 Servers from AWS g5dnxlarge, which shows in the link you provided as unrestricted.

Please also share the full log for this issue, preferably addingGST_DEBUG=3 before the deepstream-app command line.
I will soon update this ASAP

5

What do you mean upto 16?

Is your scenario that 4 channels are good, but when 5 channels are in the video, one channel cannot be detected properly? Could you reproduce that with our demo code: deepstream_yolo

6

Should I have build the model file seprately and then use the
nvdsinfer_custom_impl_Yolo, as I see only bboxParsing functions to be used.

7

Yes but the numbers in my case are 5 channels are awesome, above 5 are getting worst in detections

8

try set your batch-size same with number source: batch-size=5

9

That increases the GPU load, but no improvement, but to what I observed was, Net-scale-factor when reduced to 0.0021 it showed better to one set of videos, I am sorry as I cannot share the videos of output or input. But this is the case.

10

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

What do you mean is that the results have got worse, rather than not being tested at all? Could you repoduce that with the demo I attached?

11

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.