TensorRT with ONNX model and RGB opencv data

AI & Data Science Deep Learning (Training & Inference) TensorRT
, ,
5

Hi @spolisetty and @NVES
I have prepared reproducible steps and attached all files and models here:

I use test_onny.py to compare the keras and the onnx runtime results:

onnx: [[('n01491361', 'tiger_shark', 0.79704237), ('n01494475', 'hammerhead', 0.047171276), ('n01484850', 'great_white_shark', 0.031171514), ('n02640242', 'sturgeon', 0.002908883), ('n02071294', 'killer_whale', 0.0022062678)]]
keras: [[('n01491361', 'tiger_shark', 0.79704267), ('n01494475', 'hammerhead', 0.047171205), ('n01484850', 'great_white_shark', 0.031171512), ('n02640242', 'sturgeon', 0.0029088883), ('n02071294', 'killer_whale', 0.002206256)]]

I convert the model on my jetson nano with:
sesd@nano:/usr/src/tensorrt/bin$ sudo ./trtexec --onnx=/home/sesd/target/model.onnx --explicitBatch --verbose --workspace=1024 --exportTimes=trace.json >> /home/sesd/target/trt_out.txt --saveEngine=/home/sesd/target/model.trt

It runns without any errors, see trt_out.txt. TensorRT version is: 7.1.3-1+cuda10.2
Running the trt engine in python with shark_image_net.py produces the expected output:

trt: [[('n01491361', 'tiger_shark', 0.79704326), ('n01494475', 'hammerhead', 0.047170877), ('n01484850', 'great_white_shark', 0.031171363), ('n02640242', 'sturgeon', 0.0029088575), ('n02071294', 'killer_whale', 0.0022062561)]]

However, the final result of the c++ application, shark_image_net.cpp is off:

[04/19/2021-21:32:35] [I] idx:2 prob:0.0740943
[04/19/2021-21:32:35] [I] idx:3 prob:0.763998
[04/19/2021-21:32:35] [I] idx:4 prob:0.0382723
[04/19/2021-21:32:35] [I] idx:6 prob:0.0198122

all idx to labels can be found here:

 2: 'great white shark, white shark, man-eater, man-eating shark, Carcharodon carcharias',
 3: 'tiger shark, Galeocerdo cuvieri',
 4: 'hammerhead, hammerhead shark',
 6: 'stingray',

Top 1 prediction is correct, tiger shark, but the output prob is slightly off and some other top 5 are just wrong. Is this just expected behaviour?

Again, I think there is something wrong with how I process the input. If I use another model, which is based on this example model but is more sensitify to output precicion and has two inputs, the output differs even more (see my initial post!). Another reason could be just something wrong with trt efficientnet arch, see: Output from ONNX inference and trt inference are different · Issue #1194 · NVIDIA/TensorRT · GitHub

2 Likes