Loading nvdsanalytics.py with two very slow streams, and with one very fast one

jairo.ramirez · May 2, 2023, 5:34pm

• Hardware Platform (Jetson / GPU) AGX Xavier
• DeepStream Version 6.2
• JetPack Version (valid for Jetson only) 5.1
• TensorRT Version 8.5.2.2
• Issue Type( questions, new requirements, bugs) Questions

Hello everyone and thank you for your help.

When I run nvdsanalytics.py for two video streams it takes several minutes to start the inference, attached below is the terminal output.

When I run it for 1 video stream it takes only about 20 sec to start.

Can this be improved, what is the reason for this?

Now playing...
1 :  file:///home/.../zona_ti.mp4
2 :  file:///home/.../salida_idi.mp4
Starting pipeline 

gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
gstnvtracker: Batch processing is ON
gstnvtracker: Past frame output is ON
[NvMultiObjectTracker] Initialized
WARNING: [TRT]: Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
Deserialize yoloLayer plugin: yolo
0:00:04.460180181  3229     0x2c039c60 INFO                 nvinfer gstnvinfer.cpp:680:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1909> [UID = 1]: deserialized trt engine from :/home/xavier/anterior/sources/deepstream_python_apps/apps/deepstream-yolo-TSK/model_b1_gpu0_fp32.engine
INFO: [Implicit Engine Info]: layers num: 5
0   INPUT  kFLOAT data            3x640x640       
1   OUTPUT kFLOAT num_detections  1               
2   OUTPUT kFLOAT detection_boxes 8400x4          
3   OUTPUT kFLOAT detection_scores 8400            
4   OUTPUT kFLOAT detection_classes 8400            

0:00:04.522164374  3229     0x2c039c60 WARN                 nvinfer gstnvinfer.cpp:677:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::checkBackendParams() <nvdsinfer_context_impl.cpp:1841> [UID = 1]: Backend has maxBatchSize 1 whereas 2 has been requested
0:00:04.522256330  3229     0x2c039c60 WARN                 nvinfer gstnvinfer.cpp:677:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2018> [UID = 1]: deserialized backend context :/home/xavier/anterior/sources/deepstream_python_apps/apps/deepstream-yolo-TSK/model_b1_gpu0_fp32.engine failed to match config params, trying rebuild
0:00:04.533467044  3229     0x2c039c60 INFO                 nvinfer gstnvinfer.cpp:680:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1923> [UID = 1]: Trying to create engine from model files
WARNING: [TRT]: The implicit batch dimension mode has been deprecated. Please create the network with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag whenever possible.

Loading pre-trained weights
Loading weights of yolov8n complete
Total weights read: 3192976
Building YOLO network

        Layer                         Input Shape         Output Shape        WeightPtr
(0)     conv_silu                     [3, 640, 640]       [16, 320, 320]      496
(1)     conv_silu                     [16, 320, 320]      [32, 160, 160]      5232
(2)     conv_silu                     [32, 160, 160]      [32, 160, 160]      6384
(3)     c2f_silu                      [32, 160, 160]      [48, 160, 160]      11120
(4)     conv_silu                     [48, 160, 160]      [32, 160, 160]      12784
(5)     conv_silu                     [32, 160, 160]      [64, 80, 80]        31472
(6)     conv_silu                     [64, 80, 80]        [64, 80, 80]        35824
(7)     c2f_silu                      [64, 80, 80]        [128, 80, 80]       73200
(8)     conv_silu                     [128, 80, 80]       [64, 80, 80]        81648
(9)     conv_silu                     [64, 80, 80]        [128, 40, 40]       155888
(10)    conv_silu                     [128, 40, 40]       [128, 40, 40]       172784
(11)    c2f_silu                      [128, 40, 40]       [256, 40, 40]       321264
(12)    conv_silu                     [256, 40, 40]       [128, 40, 40]       354544
(13)    conv_silu                     [128, 40, 40]       [256, 20, 20]       650480
(14)    conv_silu                     [256, 20, 20]       [256, 20, 20]       717040
(15)    c2f_silu                      [256, 20, 20]       [384, 20, 20]       1012976
(16)    conv_silu                     [384, 20, 20]       [256, 20, 20]       1112304
(17)    conv_silu                     [256, 20, 20]       [128, 20, 20]       1145584
(18)    maxpool                       [128, 20, 20]       [128, 20, 20]       -
(19)    maxpool                       [128, 20, 20]       [128, 20, 20]       -
(20)    maxpool                       [128, 20, 20]       [128, 20, 20]       -
(21)    route: 17, 18, 19, 20         -                   [512, 20, 20]       -
(22)    conv_silu                     [512, 20, 20]       [256, 20, 20]       1277680
(23)    upsample                      [256, 20, 20]       [256, 40, 40]       -
(24)    route: 23, 12                 -                   [384, 40, 40]       -
(25)    conv_silu                     [384, 40, 40]       [128, 40, 40]       1327344
(26)    c2f_silu                      [128, 40, 40]       [192, 40, 40]       1401584
(27)    conv_silu                     [192, 40, 40]       [128, 40, 40]       1426672
(28)    upsample                      [128, 40, 40]       [128, 80, 80]       -
(29)    route: 28, 8                  -                   [192, 80, 80]       -
(30)    conv_silu                     [192, 80, 80]       [64, 80, 80]        1439216
(31)    c2f_silu                      [64, 80, 80]        [96, 80, 80]        1457904
(32)    conv_silu                     [96, 80, 80]        [64, 80, 80]        1464304
(33)    conv_silu                     [64, 80, 80]        [64, 40, 40]        1501424
(34)    route: 33, 27                 -                   [192, 40, 40]       -
(35)    conv_silu                     [192, 40, 40]       [128, 40, 40]       1526512
(36)    c2f_silu                      [128, 40, 40]       [192, 40, 40]       1600752
(37)    conv_silu                     [192, 40, 40]       [128, 40, 40]       1625840
(38)    conv_silu                     [128, 40, 40]       [128, 20, 20]       1773808
(39)    route: 38, 22                 -                   [384, 20, 20]       -
(40)    conv_silu                     [384, 20, 20]       [256, 20, 20]       1873136
(41)    c2f_silu                      [256, 20, 20]       [384, 20, 20]       2169072
(42)    conv_silu                     [384, 20, 20]       [256, 20, 20]       2268400
(43)    route: 32                     -                   [64, 80, 80]        -
(44)    conv_silu                     [64, 80, 80]        [80, 80, 80]        2314800
(45)    conv_silu                     [80, 80, 80]        [80, 80, 80]        2372720
(46)    conv_linear                   [80, 80, 80]        [80, 80, 80]        2379200
(47)    route: 43                     -                   [64, 80, 80]        -
(48)    conv_silu                     [64, 80, 80]        [64, 80, 80]        2416320
(49)    conv_silu                     [64, 80, 80]        [64, 80, 80]        2453440
(50)    conv_linear                   [64, 80, 80]        [64, 80, 80]        2457600
(51)    route: 50, 46                 -                   [144, 80, 80]       -
(52)    shuffle                       [144, 80, 80]       [144, 6400]         -
(53)    route: 37                     -                   [128, 40, 40]       -
(54)    conv_silu                     [128, 40, 40]       [80, 40, 40]        2550080
(55)    conv_silu                     [80, 40, 40]        [80, 40, 40]        2608000
(56)    conv_linear                   [80, 40, 40]        [80, 40, 40]        2614480
(57)    route: 53                     -                   [128, 40, 40]       -
(58)    conv_silu                     [128, 40, 40]       [64, 40, 40]        2688464
(59)    conv_silu                     [64, 40, 40]        [64, 40, 40]        2725584
(60)    conv_linear                   [64, 40, 40]        [64, 40, 40]        2729744
(61)    route: 60, 56                 -                   [144, 40, 40]       -
(62)    shuffle                       [144, 40, 40]       [144, 1600]         -
(63)    route: 42                     -                   [256, 20, 20]       -
(64)    conv_silu                     [256, 20, 20]       [80, 20, 20]        2914384
(65)    conv_silu                     [80, 20, 20]        [80, 20, 20]        2972304
(66)    conv_linear                   [80, 20, 20]        [80, 20, 20]        2978784
(67)    route: 63                     -                   [256, 20, 20]       -
(68)    conv_silu                     [256, 20, 20]       [64, 20, 20]        3126496
(69)    conv_silu                     [64, 20, 20]        [64, 20, 20]        3163616
(70)    conv_linear                   [64, 20, 20]        [64, 20, 20]        3167776
(71)    route: 70, 66                 -                   [144, 20, 20]       -
(72)    shuffle                       [144, 20, 20]       [144, 400]          -
(73)    route: 52, 62, 72             -                   [144, 8400]         -
(74)    detect_v8                     [144, 8400]         [8400, 84]          3192976

Output YOLO blob names: 
detect_v8_75

Total number of YOLO layers: 299

Building YOLO network complete
Building the TensorRT Engine

NOTE: Number of classes mismatch, make sure to set num-detected-classes=80 in config_infer file

NOTE: letter_box is set in cfg file, make sure to set maintain-aspect-ratio=1 in config_infer file to get better accuracy

fanzh · May 4, 2023, 5:48am

at the first time, generating a new engine will cost some time, you can set model-engine-file in the configuration file, the application will use that engine directly after the first time.
please refer to opt\nvidia\deepstream\deepstream\sources\apps\sample_apps\deepstream-test1\dstest1_pgie_config.txt

jairo.ramirez · May 4, 2023, 2:33pm

Right! When I run for a stream, the first time it creates “model_b1_gpu0_fp32.engine” and then as in my config.txt I have “model-engine-file=model_b1_gpu0_fp32.engine”, the following times it doesn’t generate it and that’s why it goes fast.

But instead with two streams, the first time it creates the file “model_b2_gpu0_fp32.engine”, but what should I define in my config.txt to use this generated engine?

jairo.ramirez · May 4, 2023, 2:51pm

Fixed, I was expecting something like model-engine-2file or model-engine-two-file.

And it turns out that I only needed to add the same thing twice and it looks like this:

model-engine-file=model_b1_gpu0_fp32.engine
model-engine-file=model_b2_gpu0_fp32.engine

This works great. Thank you very much for your help. @fanzh

system · May 18, 2023, 2:52pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.