Detector1 --> cropped images --> detector 2 Application cascading in the latest back-to-back

hi:
Hello, I have referencedhttps://devtalk.nvidia.com/default/topic/1067568/deepstream-sdk/detector1-gt-cropped-images-gt-detector-2/#,The instance model is a caffe model. I only have a yolo auxiliary detector. I use the primary_detector_config.txt original file as the primary detector,My secondary detector is as follows:

[property]
gpu-id=0
net-scale-factor=1
#0=RGB, 1=BGR
model-color-format=0
custom-network-config=yolov3-tiny.cfg
model-file=yolov3-tiny.weights
#model-engine-file=model_b1_fp32.engine
labelfile-path=labels.txt
## 0=FP32, 1=INT8, 2=FP16 mode
network-mode=0
num-detected-classes=80
gie-unique-id=1
is-classifier=0
maintain-aspect-ratio=1
parse-bbox-func-name=NvDsInferParseCustomYoloV3Tiny
custom-lib-path=nvdsinfer_custom_impl_Yolo/libnvdsinfer_custom_impl_Yolo.so

报错:

root@uu-desktop:/opt/nvidia/deepstream/deepstream-4.0/sources/apps/sample_apps/back-to-back-detectors# ./back-to-back-detectors ../../../../samples/streams/test.mp4 
Now playing: ../../../../samples/streams/test.mp4

Using winsys: x11 
Opening in BLOCKING MODE 
Creating LL OSD context new
0:00:00.653026063   870   0x5586c82870 INFO                 nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<primary-nvinference-engine2> NvDsInferContext[UID 2]:initialize(): Trying to create engine from model files
Loading pre-trained weights...
Loading complete!
Total Number of weights read : 8858734
      layer               inp_size            out_size       weightPtr
(1)   conv-bn-leaky     3 x 416 x 416      16 x 416 x 416    496   
(2)   maxpool          16 x 416 x 416      16 x 208 x 208    496   
(3)   conv-bn-leaky    16 x 208 x 208      32 x 208 x 208    5232  
(4)   maxpool          32 x 208 x 208      32 x 104 x 104    5232  
(5)   conv-bn-leaky    32 x 104 x 104      64 x 104 x 104    23920 
(6)   maxpool          64 x 104 x 104      64 x  52 x  52    23920 
(7)   conv-bn-leaky    64 x  52 x  52     128 x  52 x  52    98160 
(8)   maxpool         128 x  52 x  52     128 x  26 x  26    98160 
(9)   conv-bn-leaky   128 x  26 x  26     256 x  26 x  26    394096
(10)  maxpool         256 x  26 x  26     256 x  13 x  13    394096
(11)  conv-bn-leaky   256 x  13 x  13     512 x  13 x  13    1575792
(12)  maxpool         512 x  13 x  13     512 x  13 x  13    1575792
(13)  conv-bn-leaky   512 x  13 x  13    1024 x  13 x  13    6298480
(14)  conv-bn-leaky  1024 x  13 x  13     256 x  13 x  13    6561648
(15)  conv-bn-leaky   256 x  13 x  13     512 x  13 x  13    7743344
(16)  conv-linear     512 x  13 x  13     255 x  13 x  13    7874159
(17)  yolo            255 x  13 x  13     255 x  13 x  13    7874159
(18)  route                  -            256 x  13 x  13    7874159
(19)  conv-bn-leaky   256 x  13 x  13     128 x  13 x  13    7907439
(20)  upsample        128 x  13 x  13     128 x  26 x  26        - 
(21)  route                  -            384 x  26 x  26    7907439
(22)  conv-bn-leaky   384 x  26 x  26     256 x  26 x  26    8793199
(23)  conv-linear     256 x  26 x  26     255 x  26 x  26    8858734
(24)  yolo            255 x  26 x  26     255 x  26 x  26    8858734
Output blob names :
yolo_17
yolo_24
Total number of layers: 50
Total number of layers on DLA: 0
Building the TensorRT Engine...
Building complete!
0:00:28.364973192   870   0x5586c82870 INFO                 nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<primary-nvinference-engine2> NvDsInferContext[UID 2]:generateTRTModel(): Storing the serialized cuda engine to file at /opt/nvidia/deepstream/deepstream-4.0/sources/apps/sample_apps/back-to-back-detectors/model_b1_fp32.engine
Deserialize yoloLayerV3 plugin: yolo_17
Deserialize yoloLayerV3 plugin: yolo_24
0:00:28.834257441   870   0x5586c82870 INFO                 nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<primary-nvinference-engine1> NvDsInferContext[UID 1]:initialize(): Trying to create engine from model files
0:00:28.834410880   870   0x5586c82870 WARN                 nvinfer gstnvinfer.cpp:515:gst_nvinfer_logger:<primary-nvinference-engine1> NvDsInferContext[UID 1]:generateTRTModel(): INT8 not supported by platform. Trying FP16 mode.
0:01:07.114985055   870   0x5586c82870 INFO                 nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<primary-nvinference-engine1> NvDsInferContext[UID 1]:generateTRTModel(): Storing the serialized cuda engine to file at /opt/nvidia/deepstream/deepstream-4.0/samples/models/Primary_Detector/resnet10.caffemodel_b1_fp16.engine
Running...
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
Creating LL OSD context new
Frame Number = 0 Vehicle Count = 6 Person Count = 6 Face Count = 0 License Plate Count = 0
0:01:08.032555122   870   0x5586c79f20 WARN                 nvinfer gstnvinfer.cpp:1149:convert_batch_and_push_to_input_thread:<primary-nvinference-engine2> error: NvBufSurfTransform failed with error -2 while converting buffer
0:01:08.032898928   870   0x5586c79f20 WARN                 nvinfer gstnvinfer.cpp:1830:gst_nvinfer_output_loop:<primary-nvinference-engine1> error: Internal data stream error.
0:01:08.033508908   870   0x5586c79f20 WARN                 nvinfer gstnvinfer.cpp:1830:gst_nvinfer_output_loop:<primary-nvinference-engine1> error: streaming stopped, reason error (-5)
ERROR from element primary-nvinference-engine2: NvBufSurfTransform failed with error -2 while converting buffer
Error details: /dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvinfer/gstnvinfer.cpp(1149): convert_batch_and_push_to_input_thread (): /GstPipeline:pipeline/GstNvInfer:primary-nvinference-engine2
Returned, stopping playback
0:01:08.052820395   870   0x5586c79f20 WARN                 nvinfer gstnvinfer.cpp:1149:convert_batch_and_push_to_input_thread:<primary-nvinference-engine2> error: NvBufSurfTransform failed with error -3 while converting buffer
double free or corruption (out)
Aborted (core dumped)

Hi,

It looks like some issue in the buffer:

ERROR from element primary-nvinference-engine2: NvBufSurfTransform failed with error -2 while converting buffer
Error details: /dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvinfer/gstnvinfer.cpp(1149): convert_batch_and_push_to_input_thread (): /GstPipeline:pipeline/GstNvInfer:primary-nvinference-engine2
Returned, stopping playback

Could you try to add ‘process-mode=2’ configure in your YOLO detector and try it again?

[property]
gpu-id=0
<b>process-mode=2</b>
...

Thanks.

hi AstaLLL:
The problem is not solved,

root@uu-desktop:/opt/nvidia/deepstream/deepstream-4.0/sources/apps/sample_apps/back-to-back-detectors# ./back-to-back-detectors ../../../../samples/streams/test.mp4 
Now playing: ../../../../samples/streams/test.mp4

Using winsys: x11 
Opening in BLOCKING MODE 
Creating LL OSD context new
0:00:00.714180655  3082   0x55a5182670 INFO                 nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<primary-nvinference-engine2> NvDsInferContext[UID 2]:initialize(): Trying to create engine from model files
Loading pre-trained weights...
Loading complete!
Total Number of weights read : 8858734
      layer               inp_size            out_size       weightPtr
(1)   conv-bn-leaky     3 x 416 x 416      16 x 416 x 416    496   
(2)   maxpool          16 x 416 x 416      16 x 208 x 208    496   
(3)   conv-bn-leaky    16 x 208 x 208      32 x 208 x 208    5232  
(4)   maxpool          32 x 208 x 208      32 x 104 x 104    5232  
(5)   conv-bn-leaky    32 x 104 x 104      64 x 104 x 104    23920 
(6)   maxpool          64 x 104 x 104      64 x  52 x  52    23920 
(7)   conv-bn-leaky    64 x  52 x  52     128 x  52 x  52    98160 
(8)   maxpool         128 x  52 x  52     128 x  26 x  26    98160 
(9)   conv-bn-leaky   128 x  26 x  26     256 x  26 x  26    394096
(10)  maxpool         256 x  26 x  26     256 x  13 x  13    394096
(11)  conv-bn-leaky   256 x  13 x  13     512 x  13 x  13    1575792
(12)  maxpool         512 x  13 x  13     512 x  13 x  13    1575792
(13)  conv-bn-leaky   512 x  13 x  13    1024 x  13 x  13    6298480
(14)  conv-bn-leaky  1024 x  13 x  13     256 x  13 x  13    6561648
(15)  conv-bn-leaky   256 x  13 x  13     512 x  13 x  13    7743344
(16)  conv-linear     512 x  13 x  13     255 x  13 x  13    7874159
(17)  yolo            255 x  13 x  13     255 x  13 x  13    7874159
(18)  route                  -            256 x  13 x  13    7874159
(19)  conv-bn-leaky   256 x  13 x  13     128 x  13 x  13    7907439
(20)  upsample        128 x  13 x  13     128 x  26 x  26        - 
(21)  route                  -            384 x  26 x  26    7907439
(22)  conv-bn-leaky   384 x  26 x  26     256 x  26 x  26    8793199
(23)  conv-linear     256 x  26 x  26     255 x  26 x  26    8858734
(24)  yolo            255 x  26 x  26     255 x  26 x  26    8858734
Output blob names :
yolo_17
yolo_24
Total number of layers: 50
Total number of layers on DLA: 0
Building the TensorRT Engine...
Building complete!
0:00:29.349012887  3082   0x55a5182670 INFO                 nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<primary-nvinference-engine2> NvDsInferContext[UID 2]:generateTRTModel(): Storing the serialized cuda engine to file at /opt/nvidia/deepstream/deepstream-4.0/sources/apps/sample_apps/back-to-back-detectors/model_b1_fp32.engine
Deserialize yoloLayerV3 plugin: yolo_17
Deserialize yoloLayerV3 plugin: yolo_24
0:00:29.808689508  3082   0x55a5182670 INFO                 nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<primary-nvinference-engine1> NvDsInferContext[UID 1]:initialize(): Trying to create engine from model files
0:00:29.808830915  3082   0x55a5182670 WARN                 nvinfer gstnvinfer.cpp:515:gst_nvinfer_logger:<primary-nvinference-engine1> NvDsInferContext[UID 1]:generateTRTModel(): INT8 not supported by platform. Trying FP16 mode.
0:01:08.540191801  3082   0x55a5182670 INFO                 nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<primary-nvinference-engine1> NvDsInferContext[UID 1]:generateTRTModel(): Storing the serialized cuda engine to file at /opt/nvidia/deepstream/deepstream-4.0/samples/models/Primary_Detector/resnet10.caffemodel_b1_fp16.engine
Running...
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
Creating LL OSD context new
Frame Number = 0 Vehicle Count = 6 Person Count = 6 Face Count = 0 License Plate Count = 0
0:01:09.468900354  3082   0x55a5179f20 WARN                 nvinfer gstnvinfer.cpp:1149:convert_batch_and_push_to_input_thread:<primary-nvinference-engine2> error: NvBufSurfTransform failed with error -2 while converting buffer
0:01:09.469822617  3082   0x55a5179f20 WARN                 nvinfer gstnvinfer.cpp:1830:gst_nvinfer_output_loop:<primary-nvinference-engine1> error: Internal data stream error.
ERROR from element primary-nvinference-engine2: NvBufSurfTransform failed with error -2 while converting buffer
0:01:09.469968504  3082   0x55a5179f20 WARN                 nvinfer gstnvinfer.cpp:1830:gst_nvinfer_output_loop:<primary-nvinference-engine1> error: streaming stopped, reason error (-5)
Error details: /dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvinfer/gstnvinfer.cpp(1149): convert_batch_and_push_to_input_thread (): /GstPipeline:pipeline/GstNvInfer:primary-nvinference-engine2
Returned, stopping playback
0:01:09.489323239  3082   0x55a5179f20 WARN                 nvinfer gstnvinfer.cpp:1149:convert_batch_and_push_to_input_thread:<primary-nvinference-engine2> error: NvBufSurfTransform failed with error -3 while converting buffer
double free or corruption (out)
Aborted (core dumped)

Does the darknet algorithm support this? Back-to-back uses the caffe model.

hi:
Is there something wrong with my model? Can you provide yolo’s algorithm, let me test it?

Hi,

Let me give it a try.
Will get back to you asap.

Thanks.

hi AastaLLL:
I am concerned that our deepstream yolo interface only has vehicle detection applications. In objectDetect_Yolo, does the nvinfer in deepstream support back-to-back applications? Have you tried it? Can our deepstream provide a set of yolo applications for deepstream-test2, not just caffee.
Thank you

hi AastaLLL:
Any test results?

Dear AastaLLL:
     Can I provide you the yolo algorithm I used for testing? I can email you if needed

Hi,

Sorry to keep you waiting.

We are still trying to reproduce this issue on our side.
A simple reproducible from you will definitely help us to debugging.

You can share the source from private message.

Thanks.

Dear AastaLLL:
    Can the yolo algorithm you used for testing be shared with me for reference? If the nvinfer plugin is really no problem for yolo, then I can be sure that it is a problem with my darknet algorithm. Thank you very much

Hi,

Sorry for keeping you waiting.

We can reproduce this issue with our official YOLO model directly.
This issue will be passed into our internal team and will update more information with you once we got a feedback.

Thanks.

1 Like

Hi,

We found out the cause of this issue.

This is because the YOLO network resolution is 416x416.
However, there is no limitation on the size of object crops sent for secondary inference.
Jetson scaling will fail for factor of 16.

To fix this, please set input-object-min-width = 28 and input-object-min-height = 28.
We have verified that YOLO can work correctly as secondary detector with the update.

secondary_detector_config.txt

[property]
...
input-object-min-width=28
input-object-min-height=28

Thanks.

hi AastaLLL:
      Thank you for your excellent work, I can use yolo according to your method. However, it seems that there are still some problems that need to be adapted. When my deepstream runs back-to-back, the image is very stuttered, as shown below file:///C:/Users/Administrator/Documents/Tencent%20Files/374789127/FileRecv/MobileFile/IMG_3102.JPG, how do I adapt it?


yolo.zip (31.1 MB)

hi AastaLLL:
This is the yolo auxiliary detector I used. It is an algorithm to detect the lights. The algorithm input is a picture of the size of the vehicle. Please help me to see what is the reason, thank you very much!

Hi,

Sorry for the late update.

For a customized YOLO model, you will need to update several parameter based on your setting.
Please follow our tutorial to setup the parameter:
https://docs.nvidia.com/metropolis/deepstream/Custom_YOLO_Model_in_the_DeepStream_YOLO_App.pdf

Thanks.

hi AastaLLL:
Hello, I trained according to the darknet provided by you, and set the parameters according to your setup tutorial, and our same training method can be applied to vehicle detection on tx2, but auxiliary detection cannot be applied, can you help me Test it? I have provided the corresponding algorithm in the attachment.

Hi,

Sorry for the late update.

Do you mean the attachment in comment#13?

Thanks.

hi AastaLLL:
Yes, thank you for your reply and look forward to your feedback on testing.

Hi, 374789127

Sorry for the late reply.

Have you fixed this issue?
Does custom the YOLO parameter with this document helps?

Thanks.