tensorrt and caffe output have large difference with probability of a few percent

Hello, I have a problem.

I test 100 images, and find that, for tensorrt and caffe output(such as softmax layer), although most images have very small difference, about 0.0x or 0.00x. but several images still have very large difference, about 0.x. This phenomenon occurs for tensorrt4 and tensorrt5.
So what causes this? can you give some advises?

As the image preprocessing use opencv for tensorrt, so if the image preprocessing also use opencv for caffe inference, and the output will be same. Close this problem now.