Classifier result on onnx doesn't match Deepstream result

Accelerated Computing Intelligent Video Analytics DeepStream SDK
, , ,
#1

Hello All,

I converted two classifiers (resnet50 and mobilenetV2 ) from tenserflow .pb to onnx using this GitHub repo https://github.com/onnx/tensorflow-onnx

I compare between .onnx and .pb reults after the conversion and I got the same result,

but when I deploy Onnx model on Deepstream I got different result.
Also I got Mismatch result with the same preprocessing step in the configuration file when I run the model on Deepstream (gpu and Jetson)

• Hardware Platform (Jetson / GPU)
tested on Jetson and GPU
• DeepStream Version
5
• JetPack Version (valid for Jetson only)
4.*
• TensorRT Version
7.*
• NVIDIA GPU Driver Version (valid for GPU only)
• Issue Type( questions, new requirements, bugs)

#3

Hi,

Since Deepstream uses TensorRT as a backend inference engine.
Could you try to run your model with TensorRT to check the output first?

You can use trtexec with pre-dumped image data.

/usr/src/tensorrt/bin/trtexec --onnx=[your/model] --loadInputs=[your/input]

Thanks.

#4

Thank you for your response.
how can I use LoadInputs, what is the format needed, It is not mentioned in any doc

#5

Hi Asaldossari,

You can execute the command like this one which I used (‘data’ is the name of the input node of network):

/usr/src/tensorrt/bin/trtexec --onnx=efficientdet-d0-s.onnx --loadInputs=‘data’:o4_clip-1.jpg

I’ve got the same issue as you faced, I exported efficientdet pytorch model to onnx, and I did test by onnxruntime and found the inference/detection result of onnx is as good as that of the original pytorch model, but when I let Deepstream5 parse the efficientdet onnx with network-mode=0 (FP32 mode) and do inference by engine which is created by onnx parser, the detection result is very bad with the same image data, detection precision degraded very much. Have you resolved your issue ? thanks.

@AastaLLL could you please give some advice ? thanks.

#6

Hi, both

The input should be raw data rather than a compressed format.

Thanks.

#7

Thanks, got it, this jpg is only for testing.
As you know, when doing inference in DeepStream, the input image data is totally manipulated by nvinfer, that image should be raw data from video.
How to resolve the bad degradation in precision seen with the engine parsed out from onnx by TRT ? thanks.

#8

I tried this method, and still the result is not correct

#9

I executed trtexec to have done inference and exported output to result.json, and managed to parse the .json file to have got bboxes, the result is as bad as I saw with deepstream recognizing video. So, it looks like the cause of inference precision degradation should be in TRT’s parsing onnx and building network.

#10

Hi,

Thank you both for the clarification.
It seems the root cause is from TensorRT rather than Deepstream.

Could you share the sample that can output expected (onnxruntime?) and incorrect (TensorRT) results with us?
We want to check this with the TensorRT team.

Thanks.

#11

Hi,
I have managed to add some code to dump out our result data of running efficientdet-d0.onnx by OnnxRuntime to the json file onnxruntime_result.json with the same format as that of trtexec-result.json, which was output by running trtexec with efficientdet-d0.onnx.
And so, the raw data, which is preprocessed (BGR2RGB, scaling,padding,normalization …) for efficientdet-d0.onnx with OnnxRuntime, is also written into a file o4_clip-1_raw_data.bin, I got trtexec-result.json by runing trtexec with --loadInputs=‘data’:‘o4_clip-1_raw_data.bin’, you can also do test with it for investigation.
The final results with bbox(s) drawn on original image are also attached for your reference:
o4_clip-1-OnnxRuntime_result.jpg is the final result parsed from the output of running OnnxRuntime with efficientdet-d0.onnx, and o4_clip-1-TensorRT_result.jpg is the final result parsed from trtexec-result.json, which is the output .json file of running trtexec with efficientdet-d0.onnx. All bboxes are filtered with score_threshold=0.1 and iou_threshold=0.1.
Please note the above efficientdet-d0.onnx was not exported out with the standard efficientdet-d0.pth weight file(which was trained out with coco dataset), it was exported out with a weight file which was trained out with our own dataset, there is only one class: baggage. The original image file o4_clip-1.png is also attached for your reference.

Thanks!
eff-output-results.zip (3.6 MB)
efficientdet-d0-onnx.zip (2.1 MB)
eff-image-data-and-results.zip (3.3 MB)

#12

Sorry, for the onnx, please use this efficientdet-d0-s.zip: efficientdet-d0-s.zip (2.0 MB)

#13

Hello, any update ? thanks.

Please ignore the result seen in o4_clip-1-TensorRT_result.jpg attached last time, I forgot to scale bbox back to their actual size with the scale ratio used in image preprocessing for network, so the size and position of the two bboxes in o4_clip-1-TensorRT_result.jpg are not right.
Now I have corrected this error and did more testing and collected the image raw data and the results got by OnnxRutime vs trtexec, you can see, sometime onnx running with trtexec could recognize the same targets, but the score is much smaller than that got by onnx running with OnnxRuntime, and sometime the score for a target was under 0.1 (I set the confidence threshold to 0.1), so no bbox was drawn out (Please see o4_clip-5-TensorRT_result.jpg vs o4_clip-5-OnnxRuntime_result.jpg).

Please use RAR tools to extract the attached zip files as a whole:

effcientdet-onnx_ort-vs-trtexec.part01.zip (4 MB)
effcientdet-onnx_ort-vs-trtexec.part02.zip (4 MB)
effcientdet-onnx_ort-vs-trtexec.part03.zip (4 MB)
effcientdet-onnx_ort-vs-trtexec.part04.zip (3.8 MB)

#14

Hi,

Thanks for sharing the information.

There is a similar issue in Inference of model using tensorflow/onnxruntime and TensorRT gives different result and we are checking it right now.

Will update here for any further progress.
Thanks.

#16

Hi,

We have confirmed that the above topic is a user bug in pre-processing.
Could you check if you are meeting the similar first?

Thanks.

#17

No, I think the cause of the precision degradation I saw is totally different, as I got the raw input data for OnnxRuntime’s run() by np.tofile(), the same raw data runs with OnnxRuntime’s run() could get much better result than that runs with trtexec. You can have a test with the raw data which I uploaded in the zip file on 5/Nov, to compare trtexec and OnnxRuntime.

#18

I have a similar issue with efficentnet and efficientdet when converting from onnx to tensorrt. In onnxruntime the predictions are fine but when doing inference with tensorrt I only get a zero tensor

#19

Hi, both

Thanks for your reporting.

We are trying to reproducing this issue with the date shared by akachen in Nov 5.
Will let you know for any progress.

Thanks.

#20

Hi, akachen

Sorry for the late.
We try to reproduce this issue with following file:

efficientdet-d0-s.zip
effcientdet-onnx_ort-vs-trtexec.part01.zip (4 MB)
effcientdet-onnx_ort-vs-trtexec.part02.zip (4 MB)
effcientdet-onnx_ort-vs-trtexec.part03.zip (4 MB)
effcientdet-onnx_ort-vs-trtexec.part04.zip (3.8 MB)

But found that we cannot extract the onnx file correctly.
Could you re-compressed it and share it with us again?

Thanks.

#21

Sorry, I forgot to attach the left parts of efficicientdet-d0-s.zip, I attach all of them here:
efficientdet-d0-s.zip (2.0 MB)
efficientdet-d0-s.z01.zip (4 MB) efficientdet-d0-s.z02.zip (4 MB)

#22

efficientdet-d0-s.z03.zip (4 MB)

After download all the above 4 files to a same directory, change the name of ''efficientdet-d0-s.z01.zip" to “efficientdet-d0-s.z01”, and do the same changes to “efficientdet-d0-s.z02.zip” and “efficientdet-d0-s.z03.zip”, then unzip efficientdet-d0-s.zip to get efficientdet-d0-s.onnx