Why does the image buffer before performing inference not match the buffer received via opencv?

[dskolesnikova]

Hardware Platform - NVIDIA GeForce GTX 1070
DeepStream Version 6.2
TensorRT Version 8.5.2-1+cuda11.8
NVIDIA GPU Driver Version 525.85.12

I noticed a difference in getting results from the tensorRT engine in nvinfer plugin when using the model in deepstream and my individual app.

Initialization of the model and deserialization are equally successful, the mean scale values are completely the same, versions of the cuda, cudnn, cudart libraries are completely the same, but there is a difference in the results in the third significant digit.

I decided to check the values of the buffer coming to inference in the deepstream, then in order to avoid problems associated with preprocessing, I decided to compare the picture before preprocessing.In order to avoid problems related to scaling, the frame size corresponds to net’s input. Source frame size 3х224х224.

To do this, I copied the image from the device’s memory to the host as follows. In /opt/nvidia/deepstream/deepstream-6.2/sources/libs/nvdsinfer/nvdsinfer_context_impl.cpp at 522 before performing the function convertFcn i have added:

std::vector<unsigned char> origindata(batchInput.inputPitch * m_NetworkInfo.height, 0); 
cudaMemcpy(origindata.data(), (unsigned char*)batchInput.inputFrames[i], sizeof(unsigned char)*origindata.size(), cudaMemcpyDeviceToHost); 
std::ofstream bfs("/home/daria/Documents/deepstream/pr1/DS-test-rtsp/Build/DSBeforePreprocess.txt"); 
for(const auto dataElem: origindata) 
{ 
bfs<<dataElem; 
};

At the same time, I performed the procedure of reading the frame buffer using opencv and python:

img = cv.imread("/home/daria/Documents/deepstream/3.jpg") 
img = np.reshape(img, (150528,)) 
np.savetxt("InputImage", img, fmt='%d', newline='|')

It should be noted here that in the resulting buffer from DeepStream there is a padding of 96 pixels/line filled with zeros. The size of the received buffer instead of the expected one (224* 224 * 3 ) corresponds to (256 * 224 * 3 ). Without taking this fact into account (ignoring padding), it should be noted that there is a difference in the buffers received.

This difference on all pixels does not exceed one in value, however, this difference eventually after preprocessing gives a difference in the resulting inference result.

Please tell me what this problem may be related to? and how to fix it…

[yuweiw]

1.The picture decoders of deepstream and opencv are different, so there will be inevitable differences after decoding.
2.You can try to use new nvstreammux:export USE_NEW_NVSTREAMMUX=yes. It won’t align automatically.
3.You can set the parameters in the nvinfer config file like below:

maintain-aspect-ratio=1
symmetric-padding=1
scaling-filter=1

[dskolesnikova]

Thank you for your recommendations. Unfortunately, adding flags did not change the buffer (there is padding even when flags

maintain-aspect-ratio=1
symmetric-padding=1

are disabled).
Using USE_NEW_NVSTREAMMUX=yes also didn’t fix the results.
Scaling Interpolation method changes the results only at 1 (NvBufSurfTransformInter_Bilinear) but the result is more incorrect then others.

[yuweiw]

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

Your picture attached is the difference of the deepstream and opencv?
Could you attach all your demo code, source file and config files?

[dskolesnikova]

1. Yes
2. Thank you for your help. Please look at the demo code. Details in the README. Archive with files on disk:
   DeepStream-test.zip - Google Drive

[yuweiw]

Source of Differences：

1. The picture docoder is different with Opencv and Deepstream.
2. The data you get from the Opencv is just the raw data after picture decoding, but the data from the deepstream nvinfer you get is scaled to the model size.

So you can try to get the raw data by addding a probe function on the nvstreammux src pad. Then compare the data with openCV.