I compare between .onnx and .pb reults after the conversion and I got the same result,
but when I deploy Onnx model on Deepstream I got different result.
Also I got Mismatch result with the same preprocessing step in the configuration file when I run the model on Deepstream (gpu and Jetson)
ā¢ Hardware Platform (Jetson / GPU)
tested on Jetson and GPU ā¢ DeepStream Version
5 ā¢ JetPack Version (valid for Jetson only)
4.* ā¢ TensorRT Version
7.* ā¢ NVIDIA GPU Driver Version (valid for GPU only) ā¢ Issue Type( questions, new requirements, bugs)
Iāve got the same issue as you faced, I exported efficientdet pytorch model to onnx, and I did test by onnxruntime and found the inference/detection result of onnx is as good as that of the original pytorch model, but when I let Deepstream5 parse the efficientdet onnx with network-mode=0 (FP32 mode) and do inference by engine which is created by onnx parser, the detection result is very bad with the same image data, detection precision degraded very much. Have you resolved your issue ? thanks.
@AastaLLL could you please give some advice ? thanks.
Thanks, got it, this jpg is only for testing.
As you know, when doing inference in DeepStream, the input image data is totally manipulated by nvinfer, that image should be raw data from video.
How to resolve the bad degradation in precision seen with the engine parsed out from onnx by TRT ? thanks.
I executed trtexec to have done inference and exported output to result.json, and managed to parse the .json file to have got bboxes, the result is as bad as I saw with deepstream recognizing video. So, it looks like the cause of inference precision degradation should be in TRTās parsing onnx and building network.
Thank you both for the clarification.
It seems the root cause is from TensorRT rather than Deepstream.
Could you share the sample that can output expected (onnxruntime?) and incorrect (TensorRT) results with us?
We want to check this with the TensorRT team.
Hi,
I have managed to add some code to dump out our result data of running efficientdet-d0.onnx by OnnxRuntime to the json file onnxruntime_result.json with the same format as that of trtexec-result.json, which was output by running trtexec with efficientdet-d0.onnx.
And so, the raw data, which is preprocessed (BGR2RGB, scaling,padding,normalization ā¦) for efficientdet-d0.onnx with OnnxRuntime, is also written into a file o4_clip-1_raw_data.bin, I got trtexec-result.json by runing trtexec with --loadInputs=ādataā:āo4_clip-1_raw_data.binā, you can also do test with it for investigation.
The final results with bbox(s) drawn on original image are also attached for your reference:
o4_clip-1-OnnxRuntime_result.jpg is the final result parsed from the output of running OnnxRuntime with efficientdet-d0.onnx, and o4_clip-1-TensorRT_result.jpg is the final result parsed from trtexec-result.json, which is the output .json file of running trtexec with efficientdet-d0.onnx. All bboxes are filtered with score_threshold=0.1 and iou_threshold=0.1.
Please note the above efficientdet-d0.onnx was not exported out with the standard efficientdet-d0.pth weight file(which was trained out with coco dataset), it was exported out with a weight file which was trained out with our own dataset, there is only one class: baggage. The original image file o4_clip-1.png is also attached for your reference.
Please ignore the result seen in o4_clip-1-TensorRT_result.jpg attached last time, I forgot to scale bbox back to their actual size with the scale ratio used in image preprocessing for network, so the size and position of the two bboxes in o4_clip-1-TensorRT_result.jpg are not right.
Now I have corrected this error and did more testing and collected the image raw data and the results got by OnnxRutime vs trtexec, you can see, sometime onnx running with trtexec could recognize the same targets, but the score is much smaller than that got by onnx running with OnnxRuntime, and sometime the score for a target was under 0.1 (I set the confidence threshold to 0.1), so no bbox was drawn out (Please see o4_clip-5-TensorRT_result.jpg vs o4_clip-5-OnnxRuntime_result.jpg).
Please use RAR tools to extract the attached zip files as a whole:
No, I think the cause of the precision degradation I saw is totally different, as I got the raw input data for OnnxRuntimeās run() by np.tofile(), the same raw data runs with OnnxRuntimeās run() could get much better result than that runs with trtexec. You can have a test with the raw data which I uploaded in the zip file on 5/Nov, to compare trtexec and OnnxRuntime.
I have a similar issue with efficentnet and efficientdet when converting from onnx to tensorrt. In onnxruntime the predictions are fine but when doing inference with tensorrt I only get a zero tensor
After download all the above 4 files to a same directory, change the name of ''efficientdet-d0-s.z01.zip" to āefficientdet-d0-s.z01ā, and do the same changes to āefficientdet-d0-s.z02.zipā and āefficientdet-d0-s.z03.zipā, then unzip efficientdet-d0-s.zip to get efficientdet-d0-s.onnx