Tensorrt C++ inference for custom onnx model gives random results

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Environment

Jetson TX2
TensorRT 8.2.1-1+cuda10.2

I have a custom TensorFlow model that I’ve converted to ONNX format. The following Python code works correctly for inference:

import cv2
import onnxruntime as ort
import numpy as np


def prepare_image(image, size=(100, 100)):
  image = cv2.resize(image, size)
  image = image.astype(np.float32)
  image = image / 255.0
  image = image[np.newaxis, :, :, np.newaxis] # Shape: [1, 100, 100, 1]
  return image

def run_inference(model, image):
  input_name = model.get_inputs()[0].name
  output_name = model.get_outputs()[0].name
   
  results = model.run([output_name], {input_name: image})[0]
  return results

image_path = 'test/100.png'
model_path = 'model.onnx'

image = cv2.imread(image_path, cv2.IMREAD_GRAYSCALE)
model = ort.InferenceSession(model_path)

prepared_image = prepare_image(image)
inference_results = run_inference(model, prepared_image)

print("Inference Results:", inference_results)`

However, I need to do this in C++. I’ve been trying for several days, but I’m getting completely random inference results for different input data. I tried analyzing the sampleOnnxMnist example. I suspect my error is in the processInput function, which originally looked like this:

bool SampleOnnxMNIST::processInput(const samplesCommon::BufferManager &buffers)
{
    const int inputH = 28;
    const int inputW = 28;

    // Read a random digit file
    srand(unsigned(time(nullptr)));
    std::vector<uint8_t> fileData(inputH * inputW);
    mNumber = (rand() % 10);
    readPGMFile(locateFile(std::to_string(mNumber) + ".pgm", mParams.dataDirs), fileData.data(), inputH, inputW);

    float *hostDataBuffer = static_cast<float *>(buffers.getHostBuffer(mParams.inputTensorNames[0]));
    for (int i = 0; i < inputH * inputW; i++)
    {
        hostDataBuffer[i] = float(fileData[i] / 255.0);
    }

    return true;
}

My modified version:

bool SampleOnnxMNIST::processInput(const samplesCommon::BufferManager &buffers)
{
    const std::string imagePath = "../data/99.png";
    const int inputH = 100;
    const int inputW = 100;

    Mat image = imread(imagePath, IMREAD_GRAYSCALE);

    Mat convertedImage;
    image.convertTo(convertedImage, CV_32F, 1.0 / 255.0);

    float *hostDataBuffer = static_cast<float *>(buffers.getHostBuffer(mParams.inputTensorNames[0]));

    for (int row = 0; row < inputH; ++row)
    {
        for (int col = 0; col < inputW; ++col)
        {
            hostDataBuffer[row * inputW + col] = convertedImage.at<float>(row, col);
        }
    }

    return true;
}

Can someone suggest a fix or provide a relevant link? I’ve searched extensively online without finding a suitable solution.

2

Hi @temarco ,
Can you please share the onnx model with us?

Thanks
Aakanksha