How to evaluate Resnet-10 model from the DeepStream reference application

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Hi

I have trained FasterRCNN model and I want to compare it(based on custom test dataset) to Resnet-10 model from the DeepStream reference application. I successfully started Resnet-10 model with OpenCV but I don’t know how to parse its output. I tried several formats but they are not working for this net.

Are there some docs about the model output format and how to parse it ?

2

Hi,

Have you tried deestream-app?
There are some resnet10-based config. Suppose you can use it directly.

For example: ${deepstream_sdk_on_jetson}/configs/deepstream-app/config_infer_primary.txt

Thanks.

3

Hi
I tried deepstream app and it’s working fine. I am trying to run the Resnet10 model separately from DeepStream. I found that output layers from this network are “conv2d_bbox;conv2d_cov/Sigmoid” (according to the config_infer_primary.txt). I want to extract bounding boxes and probabilities from them. But I cannot understand their structure.
Is this information is documented somewhere ?
Or there is a way to use this config outside of deepstream ?

Thanks

4

Hi everyone. Is there no documentation or example code showing how the results of the RestNet10 from the deepstream example ist sturcured?

We would like to compare detection quality of this net with our test images to see how good it works.

5

Hi,

Deepstream document can be found here:
https://developer.nvidia.com/embedded/deepstream-jetson-3-0-documentation

You can check the deepstream-app source code first:
${deepstream_sdk_on_jetson}/sources/apps/sample_apps/deepstream-app/

Thanks.

6

Hi,

I already looked in the docs and deepstream-app code several times. What I found from there is that we pass the network and it’s configuration to nvinfer plugin. The plugin evaluate the net, parse the results and fill Metadata structure which is the ouput from the plugin.
I am interested in ‘how nvinfer evaluates the ResNet10 model?’ and ‘how it parse the output?’. With this knowledge I will be able to evaluate the model outside of deepstream-app and I can compare results to ours.

From the resnet10.prototxt file we can see that network accept input with shape (batchsize=1, channels=3, height=368, width=640) and the net has 2 output layers. First one is with shape (batchsize=1,4,23,40) and the second one - (batchsize=1,16,23,40). First one probably contains some probabilities/confidences, the second one - bboxes. I don’t know how to parse this 2 output matrices and can’t find any information related to that.
I am new in this area and maybe I am missing something

Thanks

7

Hi,

nvinfer is our TensorRT library so you can find some informaiton in this document:
[url]https://docs.nvidia.com/deeplearning/sdk/tensorrt-developer-guide/index.html#overview[/url]

There are layer level profiling and you can find the corresponding information here:
[url]Developer Guide :: NVIDIA Deep Learning TensorRT Documentation

For output, this sample may help:
[url]https://github.com/NVIDIA-AI-IOT/deepstream_reference_apps/blob/master/yolo/samples/objectDetector_YoloV3/nvdsinfer_custom_impl_YoloV3/nvdsparsebbox_YoloV3.cpp[/url]

Thanks.

8

Sorry for the late response.
I will dig deeper in the docs.
All this stuff is new for me and I should read and experiment more.

Thanks for the information

9

Hi nikki_dzhurov,

Have you found anything about the custom nvinfer plugin for ResNet10, I’m also interested in evaluating this model output. Thanks.

10

Hi soncaoai,

Instead of trying to evaluate the resnet10 caffe model, I tried different approach. I created very simple gstreamer plugin that extracts information from output metadata of the nvinfer plugin.
In Deepstream 4.0 some of the source code of nvinfer plugin came with the SDK - but it’s not much because it’s is using other closed lib at the end.

Good luck

11

Hi,

If I use the FasterRCNN Caffe model with training for custom classes, instead of resnet10 which the default SKD uses, then how much of a performance improvement can I expect w.r.t the mapping score?

There will be accuracy downgrades if switch to a 4 layer model of the RESNET, but is there a way to profile the output from the pipeline streams?

12

Hi cataluna84,

Please open a new topic for your issue. Thanks

13

Hi AastaLLL,

The resnet10.prototxt files are different in your repository https://github.com/AastaNV/DeepStream and what comes with Deepstream 4.0, I guess the model is upgraded.

My question is that is the parser still valid that is given in your repository for the Deepstream 4.0 resnet10 model.
Actually I want to make python parser for the resnet10 model.

Thanks.

14

Hi maniKTL,

Please open a new topic for your issue. Thanks