Loading nvdsanalytics.py with two very slow streams, and with one very fast one

jairo.ramirez · May 2, 2023, 5:34pm

• Hardware Platform (Jetson / GPU) AGX Xavier
• DeepStream Version 6.2
• JetPack Version (valid for Jetson only) 5.1
• TensorRT Version 8.5.2.2
• Issue Type( questions, new requirements, bugs) Questions

Hello everyone and thank you for your help.

When I run nvdsanalytics.py for two video streams it takes several minutes to start the inference, attached below is the terminal output.

When I run it for 1 video stream it takes only about 20 sec to start.

Can this be improved, what is the reason for this?

Now playing...
1 :  file:///home/.../zona_ti.mp4
2 :  file:///home/.../salida_idi.mp4
Starting pipeline 

gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
gstnvtracker: Batch processing is ON
gstnvtracker: Past frame output is ON
[NvMultiObjectTracker] Initialized
WARNING: [TRT]: Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
Deserialize yoloLayer plugin: yolo
0:00:04.460180181  3229     0x2c039c60 INFO                 nvinfer gstnvinfer.cpp:680:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1909> [UID = 1]: deserialized trt engine from :/home/xavier/anterior/sources/deepstream_python_apps/apps/deepstream-yolo-TSK/model_b1_gpu0_fp32.engine
INFO: [Implicit Engine Info]: layers num: 5
0   INPUT  kFLOAT data            3x640x640       
1   OUTPUT kFLOAT num_detections  1               
2   OUTPUT kFLOAT detection_boxes 8400x4          
3   OUTPUT kFLOAT detection_scores 8400            
4   OUTPUT kFLOAT detection_classes 8400            

0:00:04.522164374  3229     0x2c039c60 WARN                 nvinfer gstnvinfer.cpp:677:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::checkBackendParams() <nvdsinfer_context_impl.cpp:1841> [UID = 1]: Backend has maxBatchSize 1 whereas 2 has been requested
0:00:04.522256330  3229     0x2c039c60 WARN                 nvinfer gstnvinfer.cpp:677:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2018> [UID = 1]: deserialized backend context :/home/xavier/anterior/sources/deepstream_python_apps/apps/deepstream-yolo-TSK/model_b1_gpu0_fp32.engine failed to match config params, trying rebuild
0:00:04.533467044  3229     0x2c039c60 INFO                 nvinfer gstnvinfer.cpp:680:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1923> [UID = 1]: Trying to create engine from model files
WARNING: [TRT]: The implicit batch dimension mode has been deprecated. Please create the network with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag whenever possible.

Loading pre-trained weights
Loading weights of yolov8n complete
Total weights read: 3192976
Building YOLO network

        Layer                         Input Shape         Output Shape        WeightPtr
(0)     conv_silu                     [3, 640, 640]       [16, 320, 320]      496
(1)     conv_silu                     [16, 320, 320]      [32, 160, 160]      5232
(2)     conv_silu                     [32, 160, 160]      [32, 160, 160]      6384
(3)     c2f_silu                      [32, 160, 160]      [48, 160, 160]      11120
(4)     conv_silu                     [48, 160, 160]      [32, 160, 160]      12784
(5)     conv_silu                     [32, 160, 160]      [64, 80, 80]        31472
(6)     conv_silu                     [64, 80, 80]        [64, 80, 80]        35824
(7)     c2f_silu                      [64, 80, 80]        [128, 80, 80]       73200
(8)     conv_silu                     [128, 80, 80]       [64, 80, 80]        81648
(9)     conv_silu                     [64, 80, 80]        [128, 40, 40]       155888
(10)    conv_silu                     [128, 40, 40]       [128, 40, 40]       172784
(11)    c2f_silu                      [128, 40, 40]       [256, 40, 40]       321264
(12)    conv_silu                     [256, 40, 40]       [128, 40, 40]       354544
(13)    conv_silu                     [128, 40, 40]       [256, 20, 20]       650480
(14)    conv_silu                     [256, 20, 20]       [256, 20, 20]       717040
(15)    c2f_silu                      [256, 20, 20]       [384, 20, 20]       1012976
(16)    conv_silu                     [384, 20, 20]       [256, 20, 20]       1112304
(17)    conv_silu                     [256, 20, 20]       [128, 20, 20]       1145584
(18)    maxpool                       [128, 20, 20]       [128, 20, 20]       -
(19)    maxpool                       [128, 20, 20]       [128, 20, 20]       -
(20)    maxpool                       [128, 20, 20]       [128, 20, 20]       -
(21)    route: 17, 18, 19, 20         -                   [512, 20, 20]       -
(22)    conv_silu                     [512, 20, 20]       [256, 20, 20]       1277680
(23)    upsample                      [256, 20, 20]       [256, 40, 40]       -
(24)    route: 23, 12                 -                   [384, 40, 40]       -
(25)    conv_silu                     [384, 40, 40]       [128, 40, 40]       1327344
(26)    c2f_silu                      [128, 40, 40]       [192, 40, 40]       1401584
(27)    conv_silu                     [192, 40, 40]       [128, 40, 40]       1426672
(28)    upsample                      [128, 40, 40]       [128, 80, 80]       -
(29)    route: 28, 8                  -                   [192, 80, 80]       -
(30)    conv_silu                     [192, 80, 80]       [64, 80, 80]        1439216
(31)    c2f_silu                      [64, 80, 80]        [96, 80, 80]        1457904
(32)    conv_silu                     [96, 80, 80]        [64, 80, 80]        1464304
(33)    conv_silu                     [64, 80, 80]        [64, 40, 40]        1501424
(34)    route: 33, 27                 -                   [192, 40, 40]       -
(35)    conv_silu                     [192, 40, 40]       [128, 40, 40]       1526512
(36)    c2f_silu                      [128, 40, 40]       [192, 40, 40]       1600752
(37)    conv_silu                     [192, 40, 40]       [128, 40, 40]       1625840
(38)    conv_silu                     [128, 40, 40]       [128, 20, 20]       1773808
(39)    route: 38, 22                 -                   [384, 20, 20]       -
(40)    conv_silu                     [384, 20, 20]       [256, 20, 20]       1873136
(41)    c2f_silu                      [256, 20, 20]       [384, 20, 20]       2169072
(42)    conv_silu                     [384, 20, 20]       [256, 20, 20]       2268400
(43)    route: 32                     -                   [64, 80, 80]        -
(44)    conv_silu                     [64, 80, 80]        [80, 80, 80]        2314800
(45)    conv_silu                     [80, 80, 80]        [80, 80, 80]        2372720
(46)    conv_linear                   [80, 80, 80]        [80, 80, 80]        2379200
(47)    route: 43                     -                   [64, 80, 80]        -
(48)    conv_silu                     [64, 80, 80]        [64, 80, 80]        2416320
(49)    conv_silu                     [64, 80, 80]        [64, 80, 80]        2453440
(50)    conv_linear                   [64, 80, 80]        [64, 80, 80]        2457600
(51)    route: 50, 46                 -                   [144, 80, 80]       -
(52)    shuffle                       [144, 80, 80]       [144, 6400]         -
(53)    route: 37                     -                   [128, 40, 40]       -
(54)    conv_silu                     [128, 40, 40]       [80, 40, 40]        2550080
(55)    conv_silu                     [80, 40, 40]        [80, 40, 40]        2608000
(56)    conv_linear                   [80, 40, 40]        [80, 40, 40]        2614480
(57)    route: 53                     -                   [128, 40, 40]       -
(58)    conv_silu                     [128, 40, 40]       [64, 40, 40]        2688464
(59)    conv_silu                     [64, 40, 40]        [64, 40, 40]        2725584
(60)    conv_linear                   [64, 40, 40]        [64, 40, 40]        2729744
(61)    route: 60, 56                 -                   [144, 40, 40]       -
(62)    shuffle                       [144, 40, 40]       [144, 1600]         -
(63)    route: 42                     -                   [256, 20, 20]       -
(64)    conv_silu                     [256, 20, 20]       [80, 20, 20]        2914384
(65)    conv_silu                     [80, 20, 20]        [80, 20, 20]        2972304
(66)    conv_linear                   [80, 20, 20]        [80, 20, 20]        2978784
(67)    route: 63                     -                   [256, 20, 20]       -
(68)    conv_silu                     [256, 20, 20]       [64, 20, 20]        3126496
(69)    conv_silu                     [64, 20, 20]        [64, 20, 20]        3163616
(70)    conv_linear                   [64, 20, 20]        [64, 20, 20]        3167776
(71)    route: 70, 66                 -                   [144, 20, 20]       -
(72)    shuffle                       [144, 20, 20]       [144, 400]          -
(73)    route: 52, 62, 72             -                   [144, 8400]         -
(74)    detect_v8                     [144, 8400]         [8400, 84]          3192976

Output YOLO blob names: 
detect_v8_75

Total number of YOLO layers: 299

Building YOLO network complete
Building the TensorRT Engine

NOTE: Number of classes mismatch, make sure to set num-detected-classes=80 in config_infer file

NOTE: letter_box is set in cfg file, make sure to set maintain-aspect-ratio=1 in config_infer file to get better accuracy

fanzh · May 4, 2023, 5:48am

at the first time, generating a new engine will cost some time, you can set model-engine-file in the configuration file, the application will use that engine directly after the first time.
please refer to opt\nvidia\deepstream\deepstream\sources\apps\sample_apps\deepstream-test1\dstest1_pgie_config.txt

jairo.ramirez · May 4, 2023, 2:33pm

Right! When I run for a stream, the first time it creates “model_b1_gpu0_fp32.engine” and then as in my config.txt I have “model-engine-file=model_b1_gpu0_fp32.engine”, the following times it doesn’t generate it and that’s why it goes fast.

But instead with two streams, the first time it creates the file “model_b2_gpu0_fp32.engine”, but what should I define in my config.txt to use this generated engine?

jairo.ramirez · May 4, 2023, 2:51pm

Fixed, I was expecting something like model-engine-2file or model-engine-two-file.

And it turns out that I only needed to add the same thing twice and it looks like this:

model-engine-file=model_b1_gpu0_fp32.engine
model-engine-file=model_b2_gpu0_fp32.engine

This works great. Thank you very much for your help. @fanzh

system · May 18, 2023, 2:52pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
DeepStream-Yolo:Can't generate engine file but run successfully DeepStream SDK	3	738	December 15, 2022
Deepstream Yolo output is not generating DeepStream SDK	4	370	January 6, 2023
What kind of hardware rigs can support 100+ videos analytics using deepstream? DeepStream SDK hw	30	1800	October 12, 2021
YoloV3 Deepstream SDK 4.0 performance on Jetson Nano DeepStream SDK	14	4448	October 12, 2021
Unable to start Yolo8 deepstream with MJpeg AVI DeepStream SDK jetson-inference , gstreamer	14	850	September 8, 2023
Deepstrem 5.0 Python yolo DeepStream SDK	16	2659	October 12, 2021
Deepstream deployment of yolov3 with low FPS DeepStream SDK	19	1205	October 12, 2021
Detector1 --> cropped images --> detector 2 Application cascading in the latest back-to-back DeepStream SDK nvbugs	21	1457	October 12, 2021
tx2+deepstream4.0+yolov3 DeepStream SDK	2	404	October 12, 2021
Deepstream SDK for Yolov4 losing detections after 5 streams DeepStream SDK tensorrt , cuda	9	378	October 10, 2023

Loading nvdsanalytics.py with two very slow streams, and with one very fast one

Related topics